UWB at S em E val-2020 Task 1: Lexical Semantic Change Detection (2020)

Abstract: In this paper, we describe our method for detection of lexical semantic change, i.e., word sense changes over time. We examine semantic differences between specific words in two corpora, chosen from different time periods, for English, German, Latin, and Swedish. Our method was created for the SemEval 2020 Task 1: Unsupervised Lexical Semantic Change Detection. We ranked 1st in Sub-task 1: binary change detection, and 4th in Sub-task 2: ranked change detection. We present our method which is completely unsupervised and language independent. It consists of preparing a semantic vector space for each corpus, earlier and later; computing a linear transformation between earlier and later spaces, using Canonical Correlation Analysis and orthogonal transformation;and measuring the cosines between the transformed vector for the target word from the earlier corpus and the vector for the target word in the later corpus.

## introduction language evolves with time. new words appear, old words fall out of use, the meanings of some words shift. the culture changes as well as the expected audience of the printed word. there are changes in topics, in syntax, in presentation structure. reading the natural philosophy musings of aristocratic amateurs from the eighteenth century, and comparing with a monograph from the nineteenth century, or a medical study from the twentieth century, we can observe differences in many dimensions, some of which seem hard to study. changes in word senses are both a visible and a tractable part of language evolution. computational methods for researching words' stories have the potential of helping us understand this small corner of linguistic evolution. the tools for measuring these diachronic semantic shifts might also be useful for measuring whether the same word is used in different ways in synchronic documents. the task of finding word sense changes over time is called diachronic lexical semantic change (lsc) detection. the task is getting more attention in recent years @xcite @xcite . there is also the synchronic lsc task, which aims to identify domain-specific changes of word senses compared to general-language usage @xcite . @xcite provides a comprehensive survey of techniques for the lsc task, as does @xcite . @xcite evaluated available approaches for lsc detection using the durel dataset @xcite . some of the methodologies for finding time-sensitive meanings were borrowed from information retrieval techniques in the first place. according to @xcite , there are mainly three types of approaches. (1) semantic vector spaces approaches @xcite @xcite @xcite @xcite represent each word with two vectors for two different time periods. the change of meaning is then measured by the cosine distance between the two vectors. (2) topic modeling approaches @xcite @xcite @xcite @xcite ) estimate a probability distribution of words over their different senses, i.e., topics. (3) clustering models @xcite are used to cluster words into clusters representing different senses. we participated in the semeval 2020 task 1: unsupervised lexical semantic change detection @xcite competition. in this paper, we describe our solution and submitted systems for this competition. the task consists of two sub-tasks, a binary classification task (sub-task 1) and a ranking task (sub-task 2), which involve comparing usage of target words between two lemmatized corpora, each drawn from documents from a different time period for four languages: english, german, latin, and swedish. for both sub-tasks, only the target words and two corpora for each language were provided by organizers, no annotated data. the task is intended to be solved in a completely unsupervised way. in the binary classification task, the goal is for two given corpora c 1 and c 2 (for time t 1 and t 2 ) and for a set of target words, decide which of these words changed or did not change their sense (semantic) between t 1 and t 2 . change of sense is whether the word lost or gained any sense between the two periods (corpora). the objective of the sub-task 2 is for two given corpora c 1 and c 2 , rank a set of target words according to their degree of lexical semantic change between t 1 and t 2 . a higher rank means a stronger change. target words are the same for both sub-tasks. because language is evolving, expressions, words, and sentence constructions in two corpora from different time periods about the same topic will be written in languages that are quite similar but slightly different. they will share the majority of their words, grammar, and syntax. the main idea behind our solution is that we treat each pair of corpora c 1 and c 2 as different languages l 1 and l 2 even though that text from both corpora is written in the same language. we believe that these two languages l 1 and l 2 will be extremely similar in all aspects, including semantic. we train separate semantic space for each corpus and subsequently, we map these two spaces into one common cross-lingual space. we use methods for cross-lingual mapping @xcite @xcite @xcite and thanks to the large similarity between l 1 and l 2 the quality of transformation should be high. we compute cosine similarity to classify and rank the target words, see section 3 for details. our systems foot_0 ranked 1 st out of 33 teams in sub-task 1 with an average accuracy of 0.687, and 4 th out of 32 teams in sub-task 2 with an average spearman's rank correlation of 0.481. ## data the corpora are drawn from several sources, described in @xcite . lemmatization reduces the vocabulary so that there are more examples of each word. it also introduces ambiguity; the decisions to add a pos tag to english target words and retain german noun capitalization shows that the organizers were aware of this problem. ## system description first, we train two semantic spaces from corpus c 1 and c 2 . we represent the semantic spaces by a matrix x s (i.e., a source space s) and a matrix x t (i.e, a target space t) foot_1 using word2vec skip-gram with negative sampling @xcite . we perform a cross-lingual mapping of the two vector spaces, getting two matrices xs and xt projected into a shared space. we select two methods for the crosslingual mapping canonical correlation analysis (cca) using the implementation from @xcite and a modification of the orthogonal transformation from vecmap @xcite . both of these methods are linear transformations. in our case, the transformation can be written as follows: the submissions that use cca are referred to as cca-nn, cca-bin, cca-nn-r and cca-bin-r where the -r part means that the source and target spaces are reversed, see section 4. the -nn and -bin parts refer to a type of threshold used only in the sub-task 1, see section 3.1. thus, in sub-task 2, there is no difference for the following pairs of submissions: cca-nncca-bin and cca-nn-rcca-bin-r. in the case of the orthogonal transformation, the submissions are referred to as ort & uns. we use orthogonal transformation with a supervised seed dictionary consisting of all words common to both semantic spaces. (ort). the transformation matrix w @xmath0: finally in all transformation methods, for each word w i from the set of target words t , we select its corresponding vectors v s w i and v t w i from matrices xs and xt , respectively (v s w @xmath1, and we compute cosine similarity between these two vectors. the cosine similarity is then used to generate an output for each sub-task. for sub-task 2, we compute the degree of change for word w i as 1 -cos(v s w i , v t w i ). the settings of hyper-parameters for both methods give us several combinations, see sections 4 and 5. ## experimental setup to obtain the semantic spaces, we employ skip-gram with negative sampling @xcite . we use the gensim framework @xcite for training the semantic spaces. for the final submission, we trained the semantic spaces with 100 dimensions for five iterations with five negative samples and window size set to five. each word has to appear at least five times in the corpus to be used in the training. for all ccasubmissions, we build the translation dictionary for the cross-lingual transformation of the two spaces by removing the target words from the intersection of their vocabularies. for cca-nn-r and cca-bin-r we change the direction of the cross-lingual transformation. the initial setup for the transformation is that the source space is the space of the older corpus c 1 (represented by the matrix x s ), and the target space is the semantic space of the later corpus c 2 (represented by the matrix x t ). we reversed the order, and we use the matrix x t as the source space, which is transformed into semantic space (matrix x s ) of the older corpus, i.e. into the original source space. the two other methods are orthogonal transformations with identical words as the seed dictionary (ort) and unsupervised transformation (uns) from the vecmap tool. in these methods, we use median similarity as the threshold for the binary task. we experimented with a separate threshold for each language (bin suffix) and with a single global threshold for all the languages (gl suffix). the sample data provided with the task did not contain any labeled development target words, so we used the durel @xcite corpus and wocc @xcite ) corpus foot_3 to validate our solutions' performance. the wocc corpus contains lemmatized german sentences from deutsches @xcite for two time periods 1750-1799 and 1850-1899 , which we used to train the semantic spaces. we evaluated our system on the durel corpus, and we achieved 0.78 of spearman's correlation rank with settings that correspond to the column named ccann-r in ## results since the threshold strategy was used only for sub-task 1, there is no difference in results for sub-task 2 in examining a different problem is syntax. the lemmatization of the corpora is no obstacle to the english reader, who can usually guess the correct form, because word-order in english (and also in swedish and german) is somewhat rigid, and word inflexions are largely replaced by separate words, which have their own lemmas. in latin, word-order is quite flexible, and different authors have different preferred schemes for emphasis and euphony. two adjacent words are not required to be part of each other's context, and the most semantically related words in a sentence may be separated precisely to emphasize them. we performed post-evaluation experiments with word embedding vector size in order to discover its effect on system performance, see figure 1 containing visualization of results for sub-task 2. it shows that the optimal size for this task is in most cases between 100 and 175 for sub-task 2, and between 75 and 125 for sub-task 1 (not shown here). we also tried using fasttext @xcite instead of word2vec skip-gram to get the semantic space, with settings that correspond to the ones we used for the final submissions. the performance using fasttext was not improved in general, but for some settings we obtained slightly better results. other experiments suggest that latin results can benefit when the context size is increased during the training of the semantic spaces. according to submitted results, it seems that cca method with reversed order (columns cca-nn-r, cca-bin-r in ## conclusion applying a threshold to semantic distance is a sensible architecture for detecting the binary semantic change in target words between two corpora. our binary-threshold strategy succeeded quite well. we did have a small advantage in that the list of target words turned out to be nearly equally divided between changed (0.57) and unchanged (0.43) words. thus, choosing thresholds assuming that the division was 50/50 was not a severe problem. our experiments also reveal the limits of a threshold strategy, as shown in