Explaining and Improving BERT Performance on Lexical Semantic Change Detection (2021)

Abstract: Type- and token-based embedding architectures are still competing in lexical semantic change detection. The recent success of type-based models in SemEval-2020 Task 1 has raised the question why the success of token-based models on a variety of other NLP tasks does not translate to our field. We investigate the influence of a range of variables on clusterings of BERT vectors and show that its low performance is largely due to orthographic information on the target word, which is encoded even in the higher layers of BERT representations. By reducing the influence of orthography we considerably improve BERT’s performance.

## introduction lexical semantic change (lsc) detection has drawn increasing attention in the past years @xcite @xcite . recently, semeval-2020 task 1 and the italian follow-up task diacr-ita provided a multi-lingual evaluation framework to compare the variety of proposed model architectures @xcite . both tasks demonstrated that type-based embeddings outperform token-based embeddings. this is surprising given that contextualised token-based approaches have achieved significant improvements over the static type-based approaches in several nlp tasks over the past years @xcite . in this study, we relate model results on lsc detection to results on the word sense disambiguation data set underlying semeval-2020 task 1. this allows us to test the performance of different methods more rigorously, and to thoroughly analyze results of clustering-based methods. we investigate the influence of a range of variables on clusterings of bert vectors and show that its low performance is largely due to orthographic information on the target word which is encoded even in the higher layers of bert representations. by reducing the influence of orthography on the target word while keeping the rest of the input in its natural form we considerably improve bert's performance. ## related work traditional approaches for lsc detection are typebased @xcite . this means that not every word occurrence is considered individually (token-based); instead, a general vector representation that summarizes every occurrence of a word (including polysemous words) is created. the results of semeval-2020 task 1 and diacr-ita @xcite demonstrated that overall type-based approaches @xcite @xcite achieved better results than token-based approaches @xcite @xcite . this is surprising, however, for two main reasons: (i) contextualized token-based approaches have significantly outperformed static type-based approaches in several nlp tasks over the past years @xcite . (ii) semeval-2020 task 1 and diacr-ita both include a subtask on binary change detection that requires to discover small sets of contextualized usages with the same sense. typebased embeddings do not infer usage-based (or token-based) representations and are therefore not expected to be able to find such sets @xcite ). yet, they show better performance on binary change detection than clusterings of tokenbased embeddings @xcite . ## data and evaluation we utilize the annotated english, german and swedish datasets (eng, ger, swe) underlying semeval-2020 task 1 @xcite . each dataset contains a list of target words and a set of usages per target word from two time periods, t 1 and t 2 @xcite . for each target word, a word usage graph (wug) was annotated, where nodes represent word usages, and weights on edges represent the (median) semantic relatedness judgment of a pair of usages, as exemplified in (1) and (2) for the target word plane. the final wugs were clustered with a variation of correlation clustering @xcite (see figure 1 in appendix a, left) and split into two subgraphs representing nodes from t 1 and t 2 respectively (middle and right). clusters are interpreted as senses, and changes in clusters over time are interpreted as lexical semantic change. schlechtweg et al. then infer a binary change value b(w) for subtask 1 and a graded change value g(w) for subtask 2 from the two resulting time-specific clusterings for each target word w. the evaluation of the shared task participants only relied on the change values derived from the annotation, while the annotated usages were not released. we gained access to the data set, which enables us to relate performances in change detection to the underlying data. 1 we can also analyze the inferred clusterings with respect to bias factors, and compare their influence on inferred vs. gold clusterings. a further advantage of having access to the underlying data is that it reflects more accurately the annotated change scores. in semeval-2020 task 1 the annotated usages were mixed with additional usages to create the training corpora for the shared task, possibly introducing noise on the derived change scores. ## models and measures bert bidirectional encoder representations from transformers (bert, @xcite is a transformer-based neural language model designed to find contextualised representations for text by analysing left and right contexts. the base version processes text in 12 different layers. in each layer, a contextualized token vector representation is created for every word. a layer, or a combination of multiple layers (we use the average), serves as a representation for a token. for every target word, we feed the usages from the semeval data set into bert and use the respective pre-trained cased base model to create token embeddings. 2 clustering lsc can be detected by clustering the token vectors from t 1 and t 2 into sets of usages with similar meanings, and then comparing these clusters over time (cf. @xcite . this section introduces the clustering algorithms and clustering performance measures that we used. agglomerative clustering (agl) is a hierarchical clustering algorithm starting with each element in an individual cluster. it then repeatedly merges those two clusters whose merging maximizes a predefined criterion. we use ward's method, where clusters with the lowest loss of information are merged @xcite . following @xcite and @xcite , we estimate the number of clusters k with the silhouette method (rousseeuw, 1987): we perform a cluster analysis for each 2 ≤ @xmath0. the number of clusters with the largest index is used for the final clustering. the jensen-shannon distance (jsd) measures the difference between two probability distributions @xcite . we convert two time specific clusterings into probability distribution p and q and measure their distance jsd(p, q) to obtain graded change values @xcite . if p and q are very similar, the jsd returns a value close to 0. if the distributions are very different, the jsd returns a value close to 1. spearman's rank-order correlation coefficient ρ measures the strength and the direction of the relationship between two variables @xcite by correlating the rank order of two variables. its values range from -1 to 1, where 1 denotes a perfect positive relationship between the two variables, and -1 a perfect negative relationship. 0 means that the two variables are not related. cluster bias we perform a detailed analysis on what the inferred clusters actually reflect. we test hypotheses on word form, sentence position, number of proper names and corpus. the influence strength of each of these variables on the clusters is measured by the adjusted rand index (ari) @xcite between the inferred cluster labels for each test sentence and a labeling for each test sentence derived from the respective variable. for the variable word form, we assign the same label to each use where the target word has the same orthographic form (same string). if @xmath1, then the inferred clusters contain only sentences where the target word has the same form. for sentence position each sentence receives label 0, if the target word is one of the first three words of the sentence, 2, if the target word is one of the last three words, else 1. 3 for proper names a sentence receives label 0, if no proper names are in the sentence, 1, if one proper name occurs, else 2. 4 the hypothesis that proper names may influence the clustering was suggested in @xcite . for corpora, a sentence is labeled 0, if it occurs in the first target corpus, else 1. average measures given two sets of token vectors v 1 and v 2 from t 1 and t 2 , average pairwise distance (apd) is calculated by randomly picking n vectors from both sets, calculating their pairwise cosine distances d(x, y) where @xmath2. @xcite . we determine n as the minimum size of v 1 and v 2 . apd-old/new measure the average of pairwise distances within v 1 and v 2 , respectively. they are calculated as the average distance of max. 10, 000 randomly sampled unique combinations of vectors from either v 1 or v 2 . cos is calculated as the cosine distance of the respective mean vectors for v 1 and v 2 @xcite . ## conclusion bert token representations are influenced by various factors, but most strongly by target word form. even in higher layers this influence persists. by removing the form bias we were able to considerably improve the performance across languages. although we reach comparably high performance with clustering for graded change detection in german, average measures still perform better than cluster-based approaches. the reasons for this are still unclear and should be addressed in future research. furthermore, we used bert without fine-tuning. it would be interesting to see how fine-tuning interacts with influence variables and whether it further improves performance. full t1 t2