Analysing Lexical Semantic Change with Contextualised Word Representations (2020)

Abstract: This paper presents the first unsupervised approach to lexical semantic change that makes use of contextualised word representations. We propose a novel method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements. Our extensive qualitative analysis demonstrates that our method captures a variety of synchronic and diachronic linguistic phenomena. We expect our work to inspire further research in this direction.

## introduction in the fourteenth century the words boy and girl referred respectively to a male servant and a young person of either sex (oxford english dictionary). by the fifteenth century a narrower usage had emerged for girl, designating exclusively female individuals, whereas by the sixteenth century boy had lost its servile connotation and was more broadly used to refer to any male child, becoming the masculine counterpart of girl @xcite . word meaning is indeed in constant mutation and, since correct understanding of the meaning of individual words underpins general machine reading comprehension, it has become increasingly relevant for computational linguists to detect and characterise lexical semantic change-e.g., in the form of laws of semantic change @xcite @xcite -with the aid of quantitative and reproducible evaluation procedures @xcite . most recent studies have focused on shift detection, the task of deciding whether and to what extent the concept evoked by a word has changed between time periods (e.g., @xcite @xcite @xcite @xcite . this line of work relies mainly on distributional semantic models, which produce one abstract representation for every word form. however, aggregating all senses of a word into a single representation is particularly problematic for semantic change as word meaning hardly ever shifts directly from one sense to another, but rather typically goes through polysemous stages @xcite . this limitation has motivated recent work on word sense induction across time periods @xcite @xcite @xcite . word senses, however, have shortcomings themselves as they are a discretisation of word meaning, which is continuous in nature and modulated by context to convey ad-hoc interpretations @xcite @xcite . in this work, we propose a usage-based approach to lexical semantic change, where sentential context modulates lexical meaning "on the fly" @xcite . we present a novel method that (1) exploits a pre-trained neural language model (bert; @xcite to obtain contextualised representations for every occurrence of a word of interest, (2) clusters these representations into usage types, and (3) measures change along time. more concretely, we make the following contributions: our code is available at https://github.com/ glnmario/cwr4lsc . overall, our study demonstrates the potential of using contextualised word representations for modelling and analysing lexical semantic change and opens the door to further work in this direction. in form-based approaches independent models are trained on the time intervals of a diachronic corpus and the distance between representations of the same word in different intervals is used as a semantic change score @xcite . representational coherence between word vectors across different periods can be guaranteed by incremental training procedures @xcite as well as by post hoc alignment of semantic spaces @xcite . more recent methods capture diachronic word usage by learning dynamic word embeddings that vary as a function of time @xcite @xcite . form-based models depend on a strong simplification: that a single representation is sufficient to model the different usages of a word. time-dependent representations are also created in sense-based approaches: in this case word meaning is encoded as a distribution over word senses. several bayesian models of sense change have been proposed @xcite @xcite . among these is the recent scan model @xcite , which represents (1) the meaning of a word in a time interval as a multinomial distribution over word senses and (2) word senses as probability distributions over the vocabulary. the main limitation of sense-based models is that they rely on a bag-of-words representation of context. furthermore, many of these models keep the number of senses constant across time intervals and require this number to be manually set in advance. unsupervised approaches have been proposed that do not rely on a fixed number of senses. for example, the method for novel sense identification by @xcite represents senses as clusters of short dependency-labelled contexts. like ours, this method analyses word forms within the grammatical structures they appear. however, it requires syntactically parsed diachronic corpora and focuses exclusively on nouns. none of these restrictions limit our proposed approach, which leverages neural contextualised word representations. contextualised word representations several approaches to context-sensitive word representations have been proposed in the past. schütze (1998) introduced a clustering-based disambiguation algorithm for word usage vectors, @xcite proposed creating multiple vectors for the same word and @xcite proposed to directly learn usage-specific representations based on the set of exemplary contexts within which the target word occurs. recently, neural contextualised word representations have gained widespread use in nlp, thanks to deep learning models which learn usage-dependent representations while optimising tasks such as machine translation @xcite and language modelling (dai and le, 2015, ulmfit; @xcite @xcite @xcite . state-of-the-art language models typically use stacked attention layers @xcite , they are pre-trained on a very large amount of textual data, and they can be fine-tuned for specific downstream tasks @xcite @xcite . contextualised representations have been shown to encode lexical meaning dynamically, reaching high accuracy on, e.g., the binary usage similarity judgements of the wic evaluation set (pilehvar and camacho-collados, 2019), performing on a par with state-of-the-art word sense disambiguation models @xcite , and proving useful for the supervised derivation of time-specific sense representation @xcite . in this work, we investigate the potential of contextualised word representations to detect and analyse lexical semantic change, without any lexicographic supervision. ## method: a usage-based approach to lexical semantic change we introduce a usage-based approach to lexical semantic change analysis which relies on contextualised representations of unique word occurrences (usage representations). first, given a diachronic corpus and a list of words of interest, we use the bert language model @xcite to compute usage representations for each occurrence of these words. then, we cluster all the usage representations collected for a given word into an automatically determined number of partitions (usage types) and organise them along the temporal axis. finally, we propose three metrics to quantify the degree of change undergone by a word. ## data we examine word usages in a large diachronic corpus of english, the corpus of historical american english (coha, @xcite , which covers two centuries (1810-2009) of language use and includes a variety of genres, from fiction to newspapers and popular magazines, among others. in this study, we focus on texts written between 1910 and 2009, for which a minimum of 21m words per decade is available, and discard previous decades, where data are less balanced per decade. we use the 100 words annotated with semantic shift scores by @xcite as our target words. these scores are human judgements collected by asking five annotators to quantify the degree of semantic change undertaken by each word (shown out of context) from the 1960's to the 1990's. we exclude extracellular as in coha this word only appears in three decades; all other words appear in at least 8 decades, with a minimum and maximum frequency of 191 and 108,796, respectively. we refer to the resulting set of 99 words and corresponding shift scores as the 'gems dataset' or the 'gems words', as appropriate. we collect a contextualised representation for each occurrence of these words in the second century of coha, using bert as described in section 3.2. this results in a large set of usage representations, ∼1.3m in total, which we cluster into usage types using k-means and silhouette coefficients (section 3.3). we use these usage representations and usage types in the evaluation and the analyses offered in the remaining of the paper. ## correlation with human judgements before using our proposed method to analyse language change, we assess how its key components compare with human judgements. we test whether the clustering into usage types reflects human similarity judgements (section 5.1) and to what extent the degree of change computed with our metrics correlates with shift scores provided by humans (section 5.2). ## analysis in this section, we provide an in-depth qualitative analysis of the linguistic properties that define usage types and the kinds of lexical semantic change we observe. more quantitative methods (such as taking the top n words with highest jsd, apd and ed and checking, e.g., how many cases of broadening each metric captures) are difficult to operationalise @xcite because there exist no well-established formal notions of semantic change types in the linguistic literature. to carry out this analysis, for each gems word, we identify the most representative usages in a given usage type cluster by selecting the five closest vectors to the cluster centroid, and take the five corresponding sentences as usage examples. ## conclusion we have introduced a novel approach to the analysis of lexical semantic change. to our knowledge, this is the first work that tackles this problem using neural contextualised word representations and no lexicographic supervision. we have shown that the representations and the detected semantic shifts are aligned to human interpretation, and presented a new dataset of human similarity judgements which can be used to measure said alignment. finally, through extensive qualitative analysis, we have demonstrated that our method allows us to capture a variety of synchronic and diachronic linguistic phenomena. our approach offers several advantages over previous methods: (1) it does not rely on a fixed number of word senses, (2) it captures morphosyntactic properties of word usage, and (3) it offers a more effective interpretation of lexical meaning by enabling the inspection of particular example sentences. in recent work, we have experimented with alternative ways of obtaining usage representations (using a different language model, fine-tuning, and various layer selection strategies) and we have obtained very promising results in detecting semantic change across four languages @xcite . in the future, we plan to investigate whether usage representations can provide an even finer grained account of lexical meaning and its dynamics, e.g., to automatically discriminate between different types of meaning change. we expect our work to inspire further analyses of variation and change which exploit the expressiveness of contextualised word representations.