U o B at S em E val-2020 Task 1: Automatic Identification of Novel Word Senses (2020)

Abstract: Much as the social landscape in which languages are spoken shifts, language too evolves to suit the needs of its users. Lexical semantic change analysis is a burgeoning field of semantic analysis which aims to trace changes in the meanings of words over time. This paper presents an approach to lexical semantic change detection based on Bayesian word sense induction suitable for novel word sense identification. This approach is used for a submission to SemEval-2020 Task 1, which shows the approach to be capable of the SemEval task. The same approach is also applied to a corpus gleaned from 15 years of Twitter data, the results of which are then used to identify words which may be instances of slang.

## introduction automatic lexical semantic change detection is a field of semantic analysis which aims to discern how the meanings of words change over time. as interest in the field has increased, a variety of different procedures, languages and corpora have been used, which leads to difficulty when attempting to compare different sets of results. semeval-2020 task 1, "unsupervised lexical semantic change detection" @xcite , is a task aimed at providing a single unified framework with which to compare approaches using a standardised dataset, in order to address the difficulty in attempting to compare different sets of results which arises due to the variety in procedures, languages and corpora which have previously been used. the task involves determining whether a set of target words have changed meaning in two corpora, each of which corresponds to a different time period. corpora are provided for german, english, latin and swedish. the task consists of two subtasks, one involving a binary classification of target words into words which have or have not changed meaning, the other involving a ranking of words according to degree of change. this paper describes our submission to semeval-2020 task 1 1 . to produce this submission, we use an approach based on the hierarchical dirichlet process (hdp) @xcite , an extension of latent dirichlet allocation (lda) which allows the number of senses to be unbounded. the next section presents works related to methods for identifying novel word senses, which is followed by a description of our system in section 3 and the results of the application of the system in section 4. section 5 provides an overview of our conclusions and possible directions of future work. we also use our method on a corpus constructed from data from twitter in order to explore the possibility of detecting semantic change over a shorter period of time, with a focus on where that detection can be used for identifying slang. ## related work approaches to lexical semantic change detection collect co-occurrence information on words, represented in most approaches by pointwise mutual information scores @xcite . this information may be used as the basis of an approach, as is used by @xcite and @xcite to identify semantic change over time by using context vectors. context vectors can also be used as part of a different representation. @xcite use a ppmi matrix for each time period to learn dynamic embeddings, an approach to identifying lexical change which reduces the need to align vectors that arises when using the static embeddings which are traditionally used. @xcite also use dynamic embeddings, presenting a probabilistic version of word2vec @xcite , an architecture for static embeddings. embeddings are also used by @xcite , who use domain-specific embeddings to detect domain shifts. a survey of the use of embeddings for semantic shifts detection is given in @xcite . context vector and embeddings approaches to identifying change of meaning do not have the ability to recover the word senses. @xcite give an approach to the problem of word sense induction which is based on the latent dirichlet allocation (lda) model @xcite ) of text generation. the lda model is extended from parametric to nonparametric space by @xcite with an infinite vocabulary. @xcite instead use a hierarchical dirichlet process (hdp) model @xcite for the task of word sense induction. a model based on the hdp is similarly used by @xcite , for extending the flexibility of word sense induction models in order to adapt to varying degrees of polysemy. an hdp is also used by @xcite , who apply word sense induction to dictionaries and newspapers. a similar approach is taken by @xcite , in the context of updating dictionaries. @xcite use a bayesian model for tracking gradual sense changes over time. graph-based models to finding lexical semantic change are proposed by @xcite , tracking language evolution by clustering word senses over various time periods, an approach echoed by @xcite , studying word sense change in digitised books from 1520 to 2008. the most comprehensive survey of the field of lexical semantic change detection is presented by @xcite . ## system overview the data from semeval-2020 task 1 consists of two corpora for each language, each consisting of a number of short contexts of words. though data are provided in a number of languages, only the data for english was used. of each of the two corpora, one corpus corresponds to a reference corpus and the other to a focus corpus. the reference corpus is taken to represent standard usage, and the focus corpus contains newer texts. each corpus is partitioned into short pseudo-documents, which may be treated as documents. 'pseudo-document' and 'document' will hereafter be used interchangeably. the creation of these pseudo-documents is under the assumption that each corpus takes the form of short contexts of a small number of sentences of text. all stopwords and low frequency terms are removed. an initial random set of senses is induced modelled after the generative process of the hdp, which corresponds to a partition of words inside documents, a process based on the chinese restaurant franchise (crf) representation of a two level hdp @xcite , which partitions customers at the group level figure 1 : a depiction of a chinese restaurant franchise @xcite . customers θ i sit at tables ψ t which each order menu items φ k and dishes at the top level. each document j may be considered a group. its document level chinese restaurant process (crp) generates a table index t ji for each observation i according to this provides the distribution according to which the tables for words are sampled. the ppmi values in the co-occurrences matrix are used for estimation. if the sampled value is t new , then a new table is created for the restaurant and the word assigned to it. a dish must then be selected for this table. a sample for k jt new is obtained according to this gives an assignment of each word to a cluster inside a document, and of each cluster to a corpus-level topic. senses are formed from each topic by considering the distribution of assigned words. these topics represent the senses across both corpora, inferred on pooled instances of each word. since the topics are jointly modelled, discovered topics are applicable to both corpora, which means that there is no need to reconcile these senses @xcite . words with multiple senses are represented by their position in multiple topics. once sampling has concluded, the word distributions over topics are calculated. word distributions are calculated based on the topics assigned to each observed word x ji in each document to determine a distribution over topics separately for each word for the reference and focus instances. from this, the jensen-shannon distance for the target words between the distribution in the reference and focus corpus is calculated. additionally, a method for the application to the twitter corpus is used based on the novelty diff method in @xcite , a description of which follows. the distribution of words over senses is categorised according to which instances originated in the reference corpus and which in the focus corpus. for each sense s, the novelty score is calculated as once the novelty scores have been calculated for all words, the words with novel senses can be identified as those with the highest scores of novelty diff . this will be used when applying the model to twitter data rather than the semeval task in order to identify which words have gained senses, a narrower focus which is useful in the applied context of identifying slang. ## results the model's performance on semeval-2020 task 1 is presented in in addition to the semeval competition, the same model was also used to experiment with data from twitter with the goal of determining whether the same procedure can be used for sense change over a shorter timeframe, with a particular focus on identifying slang through words which had novel senses. the corpus was constructed from a set of 500,000 tweets, spanning a period of 15 years. using the approach detailed in section 3 with the twitter data produced a set of 7 slang words. these results are the words identified as having the most difference in the senses between the two corpora according to the novelty diff metric. the model was run for 1 iteration against the twitter corpora in order to identify the lemmas determined as having most differed between the older and more recent corpus, and therefore most likely to have gained a novel sense. the 25 words with the highest value of novelty diff were extracted. the online slang dictionary urban dictionary foot_2 was used to verify whether or not each of these words had a meaning aside from its standard usage or was generated erroneously. where an identified word was a genuine instance of slang, it is listed alongside this sense as defined by urban dictionary in ## conclusions and future work this paper used bayesian word sense induction methods as the basis for identifying lexical semantic change, and produced a model for determining sense differences which was then evaluated on task 1 in the semeval-2020 workshop, which confirmed the method is appropriate to the task and works as expected. the same approach was also used with data from twitter with the goal of identifying slang, which produced a set of candidate words. whilst some of these were genuine instances of slang, a significant amount were erroneously identified, or amounted to mere lexical innovation rather than any true semantic change. the system was only tested against the english data for each subtask. future work would involve extending the system into all of the competition's languages.