Improving Diachronic Word Sense Induction with a Nonparametric B ayesian method (2023)

Abstract: Diachronic Word Sense Induction (DWSI) is the task of inducing the temporal representations of a word meaning from the context, as a set of senses and their prevalence over time. We introduce two new models for DWSI, based on topic modelling techniques: one is based on Hierarchical Dirichlet Processes (HDP), a nonparametric model; the other is based on the Dynamic Embedded Topic Model (DETM), a recent dynamic neural model. We evaluate these models against two state of the art DWSI models, using a time-stamped labelled dataset from the biomedical domain. We demonstrate that the two proposed models perform better than the state of the art. In particular, the HDP-based model drastically outperforms all the other models, including the dynamic neural model.

## introduction word meanings evolve over time. recent research works have focused on how to model such dynamic behaviour. the unsupervised task of diachronic word sense induction (dwsi) aims to capture how the meaning of a word varies continuously over time, in particular when new senses appear or old senses disappear. dwsi takes the time dimension into account and assumes that the data spans over a long continuous period of time in order to model the progressive evolution of senses across time. the dynamic behaviour of words contributes to semantic ambiguity, which is a challenge in many nlp tasks. dwsi can serve as an analytical tool to help building terminology resources and indexing documents more accurately and therefore can be beneficial for information retrieval tasks. dwsi follows the probabilistic graphical modelling approach to approximate the true meanings from the observed data. thus, in this paper, we explore the relation of dwsi with topic modelling in general and to the dynamic topic modelling techniques in particular: they both aim to discover a latent variable (sense or topic respectively) from a sequential collection of documents. despite a close relation between the tasks, topic modelling techniques are not fully explored or compared against in the current state of the art of dwsi. the state of the art of dwsi consists of only two models: @xcite and @xcite . they are both designed specifically for dwsi; both are parametric; and both are dynamic, in the sense that they both introduce a time variable into the model in order to capture the evolution of the meaning over time. @xcite propose a parametric generative model (neo) where each sense is represented as a |v |-dimensional multinomial distribution over the vocabulary v , each document is represented as a mixture of senses, and the dependency of the sense proportions on time is represented as a k-dimensional multinomial distribution over the k senses. the parameters of the model have finite dirichlet priors. a more complex model called scan @xcite allows each sense distribution over the vocabulary to evolve sequentially from adjacent time slices, as well as the senses proportion. the multinomial parameters of words and senses have logistic normal priors. the two above-mentioned models are parametric, in the sense that the number of senses (which reflects the structure of the hidden meanings in the data) is a hyper-parameter which has to be known a priori. this is not ideal given the nature of the dwsi task, which is meant to infer senses from the data. the same issue has been studied for the tasks of topic modelling and wsi; hierarchical dirichlet processes (hdp), a nonparametric hierarchical model introduced by @xcite , offer an powerful solution to this problem. hdp extends latent dirichlet allocation (lda) @xcite by placing dirichlet processes priors (dps) @xcite on the infinite-dimensional space of multinomial probability distributions. thus the number of mixture components is infinite a priori and to be inferred from the data. in contrast, lda posits a predefined number k of topics, each of which is a multinomial distribution over the vocabulary. each document has specific topic proportions from a dirichlet prior, and the topics are shared among the documents. additionally, the hdp model allows sharing topics not only among documents but also across hierarchical levels by the use of multiple dps. the intuition behind our approach relies on the fact that the hierarchical dps allow "new" senses to appear as needed, thanks to the theoretically infinite number of possible senses. therefore, the hierarchical design of dirichlet processes can capture the dynamic behaviour of the words, while inferring the optimal number of clusters directly from the data across time. word embeddings are another natural direction of potential improvement for dwsi. introduced by @xcite ; @xcite , they provide a distributed representation where words with similar meanings are close in a lower-dimensional vector space. recently, various models have been proposed which integrate word embeddings for topic modelling, however these models do not necessarily represent both words and topics using embeddings. @xcite provide an elegant solution to this problem: dynamic embedded topic model (detm) is a parametric generative model inspired by d-lda (dynamic lda) @xcite , in which each word is represented with a word embedding, and per-time topics are represented as embeddings as well. topics and topic proportions evolve sequentially from adjacent time slices. detm also directly models per-topic conditional probability of a word as the exponentiated inner product between the word embeddings and per-time topic embeddings. this results in a closer semantic correspondence between words and topics, and thus leads to better topics quality. by contrast to previous contributions in dwsi which were mostly theoretical, this paper is an empirical contribution focusing on adapting different existing topic modelling techniques to dwsi. the aim is to set the state of the art dwsi models up against two serious competitors, in order to check whether they actually fit the task of dwsi optimally. in this perspective, we adapt hdp and detm to the task of dwsi, describing our approach in §3. we test the ability of these models to detect meaning change over time using the evaluation framework proposed by @xcite , described in §4: using a large corpus of biomedical time-stamped data, including 188 ambiguous target words, we compare the proposed models with the current state of the art models neo and scan. the results, presented in §5, show that hdp-based models achieve the best results over the dataset, establishing a new state of the art for dwsi. ## related work topic modelling techniques are hierarchical probabilistic bayesian models used originally for discovering topics in a collection of documents @xcite . topic models have also been adopted for the word sense induction (wsi) task, as introduced by @xcite : word senses are treated as topics, and a short window around the target word (context) is considered instead of a full document. topic modelling techniques have been extended further to similar tasks, such as novel sense detection. novel sense detection (nsd; also called novel sense identification), introduced by @xcite , consists of determining whether a target word acquires a new sense over two independent periods of time, separated by a large gap. several authors have used hierarchical dirichlet processes (hdp) for this task over a small set of target words and/or small set of data @xcite @xcite . @xcite show in a preliminary study that hdp is also superior to lda for wsi, due to its ability to adapt to varying degrees of granularity. lau et al. @xcite extend this study using an oracle-based method to identify new senses from hdp predictions for the task of nsd, and for only five target words. sarsfield and tayyar madabushi (2020) used hdp for nsd on a larger dataset @xcite , which was proposed in a recent shared task about lexical semantic change detection (lscd), a refined version of nsd: lscd intends to answer the question of whether the meaning of a target word has changed between two independent periods of time (also separated by a large time gap). in the lscd task, methods based on static word embeddings (where the meaning of the word is represented by a single vector) achieved the highest performance. in contrast to nsd/lscd, dwsi takes the time dimension into account and thus the task of dwsi is technically broader: it aims to discriminate senses and also models the temporal dynamics of word meaning across a long continuous period of time, e.g. year by year. as a result, dwsi can track the evolution of senses, the emergence of new senses and detect the year where a new sense appears. the dwsi task is introduced independently by @xcite and @xcite ; given a target word and a time-stamped corpus, both models estimate two main parameters: the senses as distributions over words, and the senses proportions over time. @xcite extend this by also inferring the subtle meaning changes within a single sense over time, i.e. by allowing different word distributions over time for the same sense. however, these models are parametric and require the number of senses to be chosen in advance. previous approaches dealt with this issue by increasing the number of senses. for example, @xcite vary the number of senses manually for every target word, while @xcite choose an arbitrary fixed large number of senses for all the target words. additionally, evaluating and comparing such models on the dwsi task is difficult: the lack of large scale time-stamped and sense-annotated data hinders direct quantitative evaluation. the state of the art models, @xcite , were originally evaluated only qualitatively on a few hand-picked target words, with a manual investigation of the quality of the associated top words in each cluster; @xcite also evaluated their model on several indirect tasks. @xcite demonstrate that these evaluation methods are insufficient, and consequently propose a quantitative evaluation of these dwsi models based on a large set of data. in particular, they show that the senses size distribution plays a significant role in capturing the senses representations and emergence of new senses. the number of senses is clearly a crucial hyperparameter for a dwsi model, the choice of which should in theory depend on the characteristics of the data. ## conclusion and discussion in this paper we adapted two topic modelling methods to the task of dwsi and evaluated them against two state of art dwsi systems, neo and scan, using the evaluation framework proposed by @xcite . we also compared using the v-measure, and proposed an improved version of the emergence algorithm. the results show that hdp-based models are able to fit the data better than the parametric models. the results strongly show that merging hdp-dwsi clusters performs better than the detm-dwsi models and lda-like clustering, such as neo and scan. the properties of hdp make it better at accurately fitting the topics/senses, in particular when there is a high imbalance between the senses proportions, i.e. with senses smaller in size (see one could naturally assume that models designed specifically for a task would perform better on it. implicitly, the research community encourages the creation of new models and tends to reward theoretical contribution over empirical ones. thus there might be a bias in favor of designing sophisticated ad-hoc models (like neo and scan) rather than adapting existing robust models (like hdp).