Using neural topic models to track context shifts of words: a case study of COVID -related terms before and after the lockdown in A pril 2020 (2022)

Abstract: This paper explores lexical meaning changes in a new dataset, which includes tweets from before and after the COVID-related lockdown in April 2020. We use this dataset to evaluate traditional and more recent unsupervised approaches to lexical semantic change that make use of contextualized word representations based on the BERT neural language model to obtain representations of word usages. We argue that previous models that encode local representations of words cannot capture global context shifts such as the context shift of face masks since the pandemic outbreak. We experiment with neural topic models to track context shifts of words. We show that this approach can reveal textual associations of words that go beyond their lexical meaning representation. We discuss future work and how to proceed capturing the pragmatic aspect of meaning change as opposed to lexical semantic change.

## introduction various approaches have been suggested in previous research to analyze semantic change such as semantic narrowing or broadening or the appearance of new words. traditional quantitative approaches to semantic change identify meaning change by relative-frequency-based methods @xcite . another traditional method is the n-gram approach, that identifies the change in likelihood of words co-occurrence across time @xcite @xcite . while these approaches can detect and track lexical changes that rely on local lexical and syntactic differences of words in time, they cannot capture lexical changes that rely on more distant relations between words @xcite . more recent computational approaches to semantic change, have exploited a pre-trained neural language model bert @xcite to obtain contextualized representations for every occurrence of a word of interest and measure changes of semantic clusters in time @xcite . contextualized embeddings can help to detect lexical changes in case linguistic differences are encoded in non-local word relations such as in the whole construction like in here comes your coach, cinderella or in you can always go, coach @xcite . these constructions capture lexical changes by being associated with different uses of the word coach, which linguistically represent notions like (in)animacy. our goal in this paper is to capture more global relations between words than word relations in constructions. we want to capture context shifts by global representations of words that share the same paragraph or document. we use neural contextualized embeddings @xcite to generate better topics, which we consider as proxies for various word contexts, similar to constructions as proxies for word senses in previous approaches @xcite . by studying thematic relations between words in time, we find out changes in these relations. this approach is not new and has been already applied by @xcite . the authors used classical topic modeling to find out new associations of words. words associations have been described in cognitive linguistics by the notion of semantic frames @xcite . according to @xcite , every word evokes a certain frame, i.e. a conceptual structure used in human communication. words evoke certain images, feelings and (personal) experiences that can be expressed by other words used in the same context. in order to capture new associations of words in time, @xcite used topic models to track new thematic relations of words like war after september 11, 2001. the underlined assumption of their approach is that context shifts or shifts of semantic frames are closely related to topic shifts @xcite . we apply this idea to our dataset to track changes of word associations by topic modeling. our new contribution is the use of an improved version of topic models, namely neural topic models @xcite . neural topic models exploit the advantages of transformer based pre-trained language models and considerably improve the coherence of topics @xcite . a set representing the topic about fruits is considered more coherent if it contains words that represent fruits such as "apple, pear, lemon, banana, kiwi", not if it contains elements that represent other objects as well such as "apple, knife, lemon, banana, spoon." @xcite . previous work has shown that adding contextual information to neural topic models provides a significant increase in topic coherence, which is missing in bag-of-words representations @xcite . incorporation of contextualized representations can thus improve a topic model's performance. by using improved topic models, we hope to improve analyses of context shifts of words uses. in this work, we exploit a particular version of neural topic models, namely bertopic (grootendorst, 2022), which includes three ingredients: (1) a specific version of pre-trained neural language model bert @xcite ) to obtain contextualised representations, (2) additional semantic clustering of these representations, and (3) calculation of topic words on the basis of c-tf-idf, which we define in §2.5 and in the appendix b. despite other existent neural topic models such as combined tm and top2vec @xcite , we have chosen bertopic, because it is an appropriate method for modeling changes in a corpus containing short messages as it contains c-tf-idf method @xcite . this method is particularly useful for analyzing corpora comprised of short documents such as tweets. we make the following contributions: overall, our study demonstrates the potential of using neural topic models for analysing context shifts of words that have preserved their lexical meaning and are thus difficult to capture by a local analysis. the paper is organized as follows. we first start with the traditional frequency and n-gram approaches that show which words have increased in relative frequency and which words have obtained new linguistic neighbors. we then apply unsupervised approaches to lexical semantic change that make use of word embeddings. finally, we apply unsupervised topic analysis to capture thematic relations between words and context shifts. the paper finishes with a discussion and evaluation of these approaches. ## conclusion we have introduced a novel dataset that contains lexical change triggered by the covid-related outbreak. we have used this dataset to discuss different analyses capable of capturing linguistic change, namely the relative frequency analysis, the n-gram analysis and lexical change captured by word embeddings. we have shown that these analyses miss