Infinite SCAN : An Infinite Model of Diachronic Semantic Change (2022)

Abstract: In this study, we propose a Bayesian model that can jointly estimate the number of senses of words and their changes through time.The model combines a dynamic topic model on Gaussian Markov random fields with a logistic stick-breaking process that realizes Dirichlet process. In the experiments, we evaluated the proposed model in terms of interpretability, accuracy in estimating the number of senses, and tracking their changes using both artificial data and real data.We quantitatively verified that the model behaves as expected through evaluation using artificial data.Using the CCOHA corpus, we showed that our model outperforms the baseline model and investigated the semantic changes of several well-known target words.

## introduction words exhibit a range of senses depending on the context in which they are used. these senses can also change over time @xcite . for example, the word cute appeared in the early 18th century, which originally meant clever or keen-witted. by the late 19th century it signified cunning, and today, cute means attractive, pretty, or sweet @xcite . automatically capturing these semantic changes is an academic contribution to the fields of lexicology and linguistics @xcite . in recent years, many methods have been proposed for the detection of semantic changes using distributional methods @xcite . they include word embedding-based methods with alignment of word embedding spaces at different times @xcite @xcite , without alignment @xcite @xcite , and using probabilistic frameworks @xcite . word embedding-based methods describe semantic change by changes of surrounding words in semantic space. however, the learned embeddings themselves cannot account for the existence of each sense and its relative importance. in contrast, several methods have addressed these issues using a topic model architecture @xcite . these probabilistic models estimate the latent senses explicitly and consider their changes, unlike word embedding models. however, these models have a critical problem in that the number of senses is given and fixed, even though it will vary for each word, which harms the modeling of semantic change. in addition, the number of senses is rarely apparent beforehand, thus it is difficult to set it a priori. a recent method of clustering contextualized word embeddings obtained from bert can similarly track sense changes @xcite , but it does not take time evolution into account, and cannot estimate the number of senses and semantic changes jointly. therefore, to address these limitations, we propose a model that can automatically estimate the number of senses and simultaneously capture semantic changes by extending the model proposed by @xcite . to this end, we combined a dynamic topic model on gaussian markov random fields (gmrf) with a logistic stick-breaking process @xcite to realize a dirichlet process in latent euclidean space. 1 here, our work can answer the question of how many senses the word has in the context of modeling semantic change, which can be applicable to lexicography. in our experiments, we verified the performance of the proposed model in terms of the estimation accuracy of the number of senses and sense change on artificial data. then, we evaluated the model performance using real data and year example snippet 1853 the driver made room for the trunk on the top of the coach. ## experimental settings in the following experiments, we split the data (i.e. set of snippets) into period of time slice 1 to t . the time interval for year-labeled data was set to ∆@xmath0. we used vocabulary with frequency larger than 10. as model settings, we set the maximum number of senses to @xmath1, the initial sense precision parameter as κ (k) ϕ = 4 for each sense k, and the gamma parameters as @xmath2. we set a relatively large value for the word precision parameter, with κ ψ = 100.0, following @xcite . this is because we want to capture the sense change of the target word as much as possible in terms of a "shift of the sense distribution" rather than a "shift of the sense-word distribution." foot_4 finally, we ran the gibbs sampler for 2,000 iterations foot_5 and resampled κ (k) ϕ for each sense k after every 50 iterations, starting from iteration 150. ## experiments using artificial data prior to the experiments using real data, we evaluated the proposed model on artificial data to validate the model for correctly estimating arbitrary changes and the number of senses. ## experiments using real data in the experiments on real data, we firstly evaluated interpretability of the model output in the same manner as topic models (section 6.3). we further evaluated the quality of the estimation results of the proposed model in terms of the estimation of the number of senses (section 6.4) and the sense change (section 6.5). ## conclusion in this study, we proposed a statistical model that can jointly estimate the number of senses and semantic change of words by combining a dynamic topic model on gmrf with a dirichlet process. in our experiments, we demonstrated that the proposed model correctly estimates the number of word senses and semantic changes in detail, and showed that the proposed model outperforms baseline models. in the future, we would like to enhance the model by incorporating linguistic knowledge on semantic change @xcite . furthermore, we would like to work on analyzing semantic change using the proposed method such as classification of change patterns @xcite . however, if there is a time point with no data at all, the estimation is likely to fail because it violates the assumptions of the model. one solution is to adopt a relatively large value for the parameter ∆t, which controls the granularity of time shift, but for a more detailed analysis, it is necessary to use a large scale time-continuous dataset.