An Evaluation Method for Diachronic Word Sense Induction (2020)

Abstract: The task of Diachronic Word Sense Induction (DWSI) aims to identify the meaning of words from their context, taking the temporal dimension into account. In this paper we propose an evaluation method based on large-scale time-stamped annotated biomedical data, and a range of evaluation measures suited to the task. The approach is applied to two recent DWSI systems, thus demonstrating its relevance and providing an in-depth analysis of the models.

## introduction words naturally evolve through time, their meaning may encounter subtle or radical changes resulting in a variety of senses. for example, the word mouse only had the meaning of animal until it acquired a brand new sense in 1980 as computer device. but sense changes are not always so definite, a word usage may drift progressively from its original sense or be affected by historical events. a recent example of this phenomenon is the word coronavirus, which has seen a dramatic usage surge in 2020 because of the emergence of its sars-cov-2 variant. before 2020, the word coronavirus was mostly a technical term describing a family of viruses, but it is now used in the mainstream media to mean the specific sars-cov-2, the related covid19 disease or even the general health crisis and its consequences. the dynamic behaviour of words contributes to semantic ambiguity, which is a challenge in many nlp tasks. the ability to detect such changes across time could potentially benefit various applications, such as machine translation and information retrieval. in the biomedical domain, it can improve the quality of the automatic identification of senses in contexts where no complete terminology is available, such as with clinical notes, and to assist indexers who build terminology resources. recent research focused on detecting semantic shifts across time (kutuzov et al., 2018) but also diachronic word sense induction (emms and kumar jayapal, 2016). the task of diachronic word sense induction (dwsi) is similar to word sense induction (wsi) in identifying the meaning of words from their context, but also takes the temporal dimension into account. in §2 we briefly present two bayesian models that have been proposed for the dwsi task: emms and kumar jayapal (2016) proposed a model which represents the evolution of word senses in order to detect the emergence year of new senses. a different model was proposed by frermann and lapata (2016), focusing instead on capturing the subtle meaning changes within a sense over time. however evaluating such models is difficult, as the lack of large scale time-stamped data prevents direct quantitative evaluation. in this paper we introduce a method which relies on annotated biomedical data to evaluate dwsi. @xcite while the general aim of this article is the evaluation of dwsi systems across domains and genres, the biomedical domain is the only one to date which offers suitable data for the task. our approach leverages the availability of unambiguous manual annotations (and publication years) in the medline citation database in order to build a large time-stamped dataset, as detailed in §3. in §4 we introduce a range of evaluation measures which can be used to directly and quantitatively measure the performance of a dwsi system on such an annotated dataset. finally in §5 we compare the two aforementioned models using our evaluation method, which demonstrates the relevance of the approach and allows a deep analysis of the models. ## a biomedical dataset for dwsi the dwsi task requires not only target words with several senses, but also time-stamped data for every target word. the evaluation of dwsi is challenging because manual annotation of such a large amount of instances (since they span over many years) would be prohibitively costly. @xcite in this section, we propose a method to collect diachronic data for ambiguous terms in medical terminologies. ## evaluation as explained in §3, the collected dataset contains sense labels which can be used to directly evaluate a dwsi system in a reliable way. since by definition the ouput of an unsupervised clustering algorithm is unlabeled, we propose in §4.1 a method to match a gold sense with a predicted sense. thanks to this matching method, a system can be evaluated externally, in a way similar to a supervised wsd system. we propose several evaluation methods, each meant to capture the performance of a dwsi system from a different perspective. ## results and analysis in this section, we evaluate the neo and scan systems using the dataset presented in §3 and the evaluation methods defined in §4. this allows us to compare the two systems on the same grounds. additionally this rich annotated dataset allows us to provide an in-depth analysis, thus uncovering the strengths and weaknesses of the two systems. the dwsi task is unsupervised, so the whole data is used both to estimate the parameters and perform evaluation on the predictions. no parameter has been tuned at any point: the experiments are run using the systems provided by the original authors with their default parameters, except for the number of senses (the true number of senses is used for evey target), one-year time interval, and the size of the context window (10). @xcite ## conclusion and discussion we have addressed the issue of evaluating dwsi: we evaluated two models, neo and scan, directly on the task itself, independently from any extrinsic related tasks, with a large dataset collected from biomedical resources. we defined and tested various external evaluation measures. overall, neo performs significantly better in the tasks of detecting senses and the emergence of new senses, according to most of our evaluation measures. the design differences between the models and their parameters could potentially have an effect on the amount of data they require, but it turns out that the global data size has no important effect on the accuracy of either system. both systems are unable to predict the correct size of the clusters: they tend to split the data almost equally between senses irrespective of the true semantic sense represented by the context words, and this impacts the correct detection of the emergence. this issue also explains why the original studies tend to use a high number of senses in order to capture the true senses, even though this causes the clusters to be split and the appearance of "junk senses". we also find that neo performs better with larger senses while scan tends to perform better with smaller senses. this opens the perspective of combining the advantages of the two systems. we acknowledge that the data is domain-specific, however the observed biases of the systems are likely to hold across domains.