Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detection (2021)

Abstract: Lexical semantic change detection is a new and innovative research field. The optimal fine-tuning of models including pre- and post-processing is largely unclear. We optimize existing models by (i) pre-training on large corpora and refining on diachronic target corpora tackling the notorious small data problem, and (ii) applying post-processing transformations that have been shown to improve performance on synchronic tasks. Our results provide a guide for the application and optimization of lexical semantic change detection models across various learning scenarios.

## introduction in recent years lexical semantic change detection (lscd), i.e. the detection of word meaning change over time, has seen considerable developments @xcite @xcite . the recent publication of multi-lingual human-annotated evaluation data from semeval-2020 task 1 @xcite makes it now possible to compare lscd models in a variety of scenarios. the task shows a clear dominance of type-based embeddings, although these are strongly influenced by the size of training corpora. in order to mitigate this problem we propose pre-training models on large corpora and refine them on diachronic target corpora. we further improve the obtained embeddings with several post-processing transformations which have been shown to have positive effects on performance in semantic similarity and analogy tasks @xcite @xcite as well as term extraction @xcite . extensive experiments are performed on the german and english lscd datasets from semeval-2020 task 1. according to our findings, pre-training is advisable when the target corpora are small and should be done using diachronic data. we further show that pre-training on large corpora strongly interacts with vector dimensionality and propose a simple solution to avoid drastic performance drops. post-processing often yields further improvements. however, it is hard to find a reliable parameter that performs well across the board. our experiments suggest that it is possible to use simple pre-and post-processing techniques to improve the state-ofthe-art in lscd. ## related work as evident in @xcite the field of lscd is currently dominated by vector space models (vsms), which can be divided into typebased (static) @xcite and token-based (contextualized) @xcite models. prominent type-based models include lowdimensional embeddings such as global vectors @xcite and skip-gram with negative sampling @xcite . however, as these models come with the deficiency that they aggregate all senses of a word into a single representation, token-based embeddings have been proposed @xcite . according to @xcite these models can ideally capture complex characteristics of word use, and how they vary across linguistic contexts. the results of semeval-2020 task 1 @xcite , however, show that contrary to this, the token-based embedding models @xcite are heavily outperformed by the type-based ones @xcite . the sgns model was not only widely used, but also performed best among the participants in the task. this result was recently reproduced in the diacr-ita shared task @xcite @xcite . its fast implementation and combination possibilities with different alignment types further solidify sgns as the standard in lscd @xcite @xcite . hence, the embeddings used in this work are sgns-based. further increases in performance of type-based vsms can be achieved by various post-processing transformations. this has been shown for semantic similarity and analogy tasks @xcite @xcite as well as term extraction @xcite . it is still an open question whether these transformations improve performance in the special setting of lscd where we typically have several corpora and vector spaces which have to be transformed simultaneously @xcite ). an indication is given by @xcite showing that for a simple lscd model mean centering leads to consistent performance improvements on two german data sets. whether this result is reproducible on further data sets, more complex models and further post-processing techniques has not been determined yet. post-processing methods operate on information already contained in a vsm, rather than adding additional information. further semantic information can be introduced by pre-training vectors on a larger unspecific collection of text @xcite or by training a seperate matrix on such text and concatenating the two vsms @xcite . this is especially helpful for cases where only smaller specialized corpora are given. combining the information from two models is also found in @xcite , here it is used for alignment proposes. we operate similarly to @xcite with the motivation of limsopatham and collier and kutuzov and kuzmenko, as we aim to enrich a vsm prior to the training process. ## data and tasks we train sgns-based vsms on various corpora and use a word similarity task and an lscd task for evaluation. the two tasks share a common aspect: the vector representations of two words need to be compared with some metric (e.g. cosine similarity), and word pairs need to be ranked according to that metric. in the word similarity task, we have the vectors of two different words in the same vector space (w i , w j ), while for lscd we have the vectors of the same word but from different vector spaces representing different time periods (w t1 i , w t2 i ). modern data. we use two large modern english and german corpora, pukwac @xcite and sdewac @xcite to validate the post-processing methods on the word similarity task and to create pre-trained embeddings for the lscd task. pukwac and sdewac are web-crawled corpora from the .uk and .de domain respectively. resulting in fairly large corpora, 2b tokens and 750m tokens (see ## models following the popular approach taken for typebased vector space models in lscd, we combine three sub-systems: (i) creating semantic word representations, (ii) aligning them across corpora, and (iii) measuring differences between the aligned representations @xcite @xcite . alignment is needed as columns from different vector spaces may not correspond to the same coordinate axes, due to the stochastic nature of many low-dimensional word representations @xcite . additionally, we aim to refine subsystem (i) by adding pre-trained semantic word representations and using post-processing methods to improve the quality of the created semantic word representations. 1 we use sgns @xcite to create type-based word representations in combination with three different alignment methods, orthogonal procrustes (op), vector initialization (vi), and word injection (wi). the three alignment methods combined with sgns have been proven to be state-of-the-art, even when competing against token-based embeddings @xcite @xcite . cosine distance (cd) is used to measure differences between word vectors. foot_1 ## experiments for the most part, we chose common model hyperparameter settings in order to keep our results comparable to previous research @xcite @xcite . we fine-tune for different alignment methods and datasets by varying dimensionality d, window size w and number of training epochs e. foot_2 ## conclusion we tested the effects of pre-training and postprocessing on a variety of lscd models. we performed extensive experiments on a german and an english lscd dataset. according to our findings, pre-training is advisable when the target corpora are small and should be done using diachronic data. the size of the pre-training corpus is crucial, as a large number of training pairs leads to performance drops, which are probably caused by their effect on vector length. length-normalization may be used on pre-trained vectors to counteract this effect. further performance improvements may be reached by post-processing. while sot+sta yielded moderate improvements for both languages, mc+pcr showed larger improvements, but only on english. however, for neither we were able to find a reliable parameter that performed well across the board. instead, we found that a well-performing parameter value is highly dependent on the underlying matrix. both post-processing methods affect isotropy and frequency bias. the methods we tested are particularly helpful when tuning data is available, as performance can be optimized and becomes more predictable. hence, we recommend to obtain a small annotated sample of target words for the target corpora and to tune pre-training, model and post-processing parameters on the sample before performing predictions for semantic changes on unseen data. with the recent upsurge of digitized historical corpora and diachronic semantic annotation efforts @xcite @xcite @xcite this may often be a likely and feasible scenario.