SC h ME at S em E val-2020 Task 1: A Model Ensemble for Detecting Lexical Semantic Change (2020)

Abstract: This paper describes SChME (Semantic Change Detection with Model Ensemble), a method used in SemEval-2020 Task 1 on unsupervised detection of lexical semantic change. SChME uses a model ensemble combining signals distributional models (word embeddings) and word frequency where each model casts a vote indicating the probability that a word suffered semantic change according to that feature. More specifically, we combine cosine distance of word vectors combined with a neighborhood-based metric we named Mapped Neighborhood Distance (MAP), and a word frequency differential metric as input signals to our model. Additionally, we explore alignment-based methods to investigate the importance of the landmarks used in this process. Our results show evidence that the number of landmarks used for alignment has a direct impact on the predictive performance of the model. Moreover, we show that languages that suffer less semantic change tend to benefit from using a large number of landmarks, whereas languages with more semantic change benefit from a more careful choice of landmark number for alignment.

## introduction the problem of detecting lexical semantic change (lsc) consists of measuring and identifying change in word sense across time, such as in the study of language evolution, or across domains, such as determining discrepancies in word usage over specific communities @xcite . one of the greatest challenges of this problem is the difficulty of assessing and evaluating models and results, as well as the limited amount of annotated data @xcite . for that reason, the vast majority of the related work in the literature pursue this problem from an unsupervised perspective, that is, detecting semantic change without having prior knowledge of "truth". the importance of such task is manifold: to humans, it can be a powerful tool for studying language change and its cultural implications; to machines, it can be used to improve language models in downstream tasks such as unsupervised word translation, and fine-tuning of word embeddings @xcite . in this task, the goal is to develop a method for unsupervised detection of lexical semantic change over time by comparing across two corpora from different time periods in four languages: english, german, latin, and swedish @xcite . particularly, we are required to solve two sub-tasks: binary classification of semantic change (subtask 1), and semantic change ranking (subtask 2). there are many ways in which a word may change. specifically, a word w may change sense because it has been completely replaced by a synonym w s (lexical replacement), or because it gains a new meaning, in which case word w may keep or lose its previous meaning across time and domain @xcite . each type of change has its unique characteristics and may require different approaches in order to be detected. in this paper we describe a novel model ensemble method based on different features (signals) that we can extract from the text using distribution models (skip-gram word embeddings) and word frequency. our model is primarily based on features extracted from independently trained word2vec embeddings aligned with orthogonal procrustes @xcite , such as cosine distance, but also introduces two novel measures based on second-order distances and word frequency. based on the distribution of each feature, we predict the probability that a word has suffered change through an anomaly detection approach. the final decision is made by soft voting (averaging) all the probabilities. for binary classification (subtask 1) a threshold is applied to the final vote, for ranking (subtask 2), the output from the soft voting is used as the ranking prediction. our results show that second order methods and different combinations outperform the frequently used cosine distance in some subtasks and languages. furthermore, we illustrate that the methods are sensitive to the degree of change in the language. it is possible to improve performance of these methods by aligning two embeddings of the same language from different time slices on a subset of words instead of all words. this opens a new avenue of research on finding optimal words for alignment. the code for the model can be obtained at https://github.com/mgruppi/schme . ## related work most methods for detecting semantic change are based on the distributional property of word semantics. the general idea is to compute contextual information of word w in each time or domain, and apply a measure of difference or distance between the observed contexts of w. some of the first methods for detecting semantic change compute context information using a co-occurrence matrix within a pre-defined window of size l @xcite . this means that, for a vocabulary of size n, one computes a n × n matrix m where m i,j is the frequency in which word i and j co-occur within a window of l words. this often yields a highly sparse matrix m , which is typically reduced in dimensionality by techniques such as singular value decomposition (svd). once the matrices are computed, the contextual difference is computed by the cosine distance between the vectors. distributed word vector representations such as the ones obtained by the skip-gram with negative sampling (sgns) @xcite are forms of learning distributional information without the need for computing sparse co-occurrence matrices. a work by @xcite presents a method for detecting semantic change using sgns word embeddings learned from each corpora and aligned with orthogonal procrustes. the semantic change is, again, computed by the cosine distance between vectors in each time/domain. in another study @xcite , the authors introduce a measure of semantic change based on how the neighborhood of a word changes named local neighborhood change based on the number of words in common. to eliminate the need for alignment, several authors have proposed dynamic word embeddings techniques, which jointly learn distributional word representations using the assumption that words are connected across time @xcite @xcite . the main assumption in such methods is that word changes are considerably small between adjacent time stamps t 1 and t 2 , i.e. words evolve smoothly, thus word representations should be close between these periods. we argue that the assumption that all words in t 1 and t 2 should be smoothly connected through time does not always hold. this is because the corpora are aggregated over several years/decades/centuries, thus the semantic change may be drastic, and more similar to a cross-domain scenario than a diachronic one. we illustrate this by the corpora in this task and the use of a subset of landmarks for alignment that has not been investigated in the literature. ## model overview and data the data provided in this task consists of two corpora for each language, each corpus corresponding to different time periods t 1 and t 2 , as well as a list of target words for which we have to predict binary class and rank with respect to magnitude of the semantic change between t 1 and t 2 . the corpora used for each language are summarized in ## evaluation we conduct all the experiments on the data provided for semeval-2020 task 1 for all four languages. given that most of the corpora have been pre-processed with lemmatization and tokenization, our preprocessing consists of removing words whose count is less than 10, and tokenizing words at spaces. in this section we present the experiments and results for the model submitted to the task, as well as additional analysis of the model parameters. we begin by learning the distributional representations of words in each corpora using gensim's @xcite implementation of word2vec . the parameters for word2vec are: vector size @xmath0, window @xmath1, negative samples @xmath2, and minimum word count min @xmath3. next, we align the learned word vectors via op using the intersecting vocabulary as landmarks. then, we compute the distance metrics and their distributions so that we can get the vote p r(x @xmath4. finally, we apply the model ensemble to different feature configurations to predict a final score. for classification, we apply a threshold t to the model output p (x), such that the predicted class is @xmath5, and @xmath6. for ranking, the final score p (x) is used. since there was no validation data during the evaluation phase, our submissions included multiple feature and threshold settings. the feature configurations are combinations of the cosine distance (cos), mapped neighborhood distance (map), and frequency differential (freq). the applied threshold levels are {0.5, 0.75, 0.9}. our team (rpi-trust) ranked 4th place in subtask 1 with a score of 0.660, and 6th place in subtask 2 with a score of 0.427 in the evaluation phase. ## post-evaluation we evaluate our model on the provided test data in the post-evaluation phase. first, we fix a threshold of @xmath7.75, then we use different feature combinations to evaluate the performance on each language. classification results, seen in ## conclusions we presented a model for unsupervised detection of semantic change based on anomaly detection over a selection of features. schme works directly on the input corpora, not requiring language-specific pre-trained models. the model ensemble is agnostic to the feature models, which means any measure of change could be easily incorporated to it, if desired. our results show that the model parameters must be chosen carefully for each task and language. particularly, we have shown that the choice of landmarks for alignment is strictly related to the degree of change of a language. in future work, we plan on addressing this issue by developed principled ways of choosing the words to align so that the semantic change is revealed more accurately.