CIRCE at S em E val-2020 Task 1: Ensembling Context-Free and Context-Dependent Word Representations (2020)

Abstract: This paper describes the winning contribution to SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection (Subtask 2) handed in by team UG Student Intern. We present an ensemble model that makes predictions based on context-free and context-dependent word representations. The key findings are that (1) context-free word representations are a powerful and robust baseline, (2) a sentence classification objective can be used to obtain useful context-dependent word representations, and (3) combining those representations increases performance on some datasets while decreasing performance on others.

## introduction semeval-2020 task 1 poses an evaluation framework for unsupervised lexical semantic change detection (lscd). its two subtasks operate on a set of non-parallel corpus pairs from two different time periods and are evaluated against human annotations for semantic change of a subset of words. subtask 1 requires binary classification of whether or not the meaning of the words has changed. subtask 2 requires ranking the words by degree of lexical semantic change and is evaluated in spearman's rank-order correlation coefficient ρ @xcite . this paper primarily addresses subtask 2. one of the most successful methods for predicting the degree of semantic change of words is comparing context-free (static) semantic vector spaces. such models seperately induce word vectors for all words in two corpora (e.g. with word2vec @xcite ), align the resulting vector spaces and take the distance of the word vectors as a measure of semantic change @xcite . with recent advances in language model pretraining, it is now possible to extract context-dependent (contextualized) word representations for each use of a word in the two corpora by using a language model (e.g. bert @xcite ) as feature extractor. the distance between these context-dependent word representations can then be taken as a measure of semantic change @xcite . in this paper, we present a model based on context-free word representations, a model based on contextdependent word representations and an ensemble model that combines their predictions. we obtain context-free representations by following the methodology of the best model reported by @xcite , aligning sgns vectors. for context-dependent representations, we finetune bert with a sentence classification objective -predicting the time period of sentences -and extract internal representations for all words from the finetuned bert model. we show that this classification finetuning can both produce useful word representations and provide an indicator for how to parameterize the ensemble. in the evaluation phase of semeval-2020 task 1, the context-free model ranked first out of 128 contributions to subtask 2. the ensemble model performed better in one language, but significantly worse in another, causing it to be ranked fifth among all contributions. the context-dependent model ranked 69th, suggesting that it can sometimes add information to the context-free model, but is largely inadequate on its own. in the results section, we analyze the submission experiments and show that the usefulness of the context-dependent representations is linked to the bert classification accuracy. code and predictions for all models are available publicly at https://github.com/mpoemsl/circe . ## available datasets a complete dataset for lexical semantic change ranking as defined by @xcite consists of two lemmatized corpora from different time periods (t 1 and t 2 ) and a corresponding testset. this testset contains gold ranks for a subset of words as annotated by human experts. predictions are made for all target words on the basis of the two corpora and evaluated against the true ranks in the testset. in the development experiments, we validate our approach by following @xcite in evaluating our models on the diachronic testset durel @xcite (de-durel) and the synchronic testset surel @xcite (de-surel) in combination with the corresponding corpora. it should be noted that de-surel presents domain-specific rather than time-specific meaning differences, but since diachronic and synchronic lexical semantic change detection are closely related, it is still a useful development dataset. in the submission experiments, we compute our submission for the evaluation phase based on the diachronic datasets provided in semeval-2020 task 1 @xcite , which consist of an english (en-semeval), german (de-semeval), latin (ln-semeval) and swedish (sw-semeval) dataset. results are averaged over submission datasets. an overview of all datasets can be found in we present three models for semeval-2020 task 1 subtask 2: a context-free model, a context-dependent model and the ensemble model circe, which stands for classification-informed representation comparison ensemble. for subtask 1, we binarize the circe rank predictions by naively assuming that the upper half of ranks has changed while the lower half has not. ## conclusion we presented a context-free model, a context-dependent model and an ensemble model for semeval-2020 task 1 subtask 2. we showed that while the context-free model outperformed all other systems during the evaluation phase, ensembling its predictions with those of the context-dependent model leads to increased performance on the development datasets (de-surel, de-durel) and one submission dataset (ln-semeval) at the cost of decreased performance on another submission dataset (en-semeval). the submission experiments have made it clear that, although its performance in the development experiments was competitive, the context-dependent model on its own is not a reliable predictor of lexical semantic change. however, it is worth pointing out that the context-dependent representations in all cases contain at least some information that is useful for lscd, since the optimal circe weight is greater than zero in all experiments. still, the marginal improvements accomplished with this information do not justify the significant computational effort of finetuning bert for one epoch. in further research, it would be interesting to validate the methods described in this paper on additional datasets. in particular, it could be worthwhile to empirically explore the link between classification accuracy and representation usefulness. if the relation were to hold up for other domains and models with a lower computational complexity than bert, representations obtained through self-supervised classification training could be used on a whole range of other unsupervised tasks.