Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora (2020)

Abstract: The problem of comparing two bodies of text and searching for words that differ in their usage between them arises often in digital humanities and computational social science. This is commonly approached by training word embeddings on each corpus, aligning the vector spaces, and looking for words whose cosine distance in the aligned space is large. However, these methods often require extensive filtering of the vocabulary to perform well, and - as we show in this work - result in unstable, and hence less reliable, results. We propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word. The method is simple, interpretable and stable. We demonstrate its effectiveness in 9 different setups, considering different corpus splitting criteria (age, gender and profession of tweet authors, time of tweet) and different languages (English, French and Hebrew).

## introduction analyzing differences in corpora from different sources (different time periods, populations, geographic regions, news outlets, etc) is a central use case in digital humanities and computational social science. a particular methodology is to identify individual words that are used differently in the different corpora. this includes words that have their meaning changed over time periods @xcite @xcite @xcite , and words that are used differently by different populations @xcite . it is thus desired to have an automatic, robust and simple method for detecting such potential changes in word usage and surfacing them for human analysis. in this work we present such a method. ## stability a desired property of an analysis method is stability: when applied several times with slightly different conditions, we expect the method to return the same, or very similar, results. insignificant changes in the initial conditions should result in insignificant changes in the output. this increases the likelihood that the uncovered effects are real and not just artifacts of the initial conditions. recent works question the stability of word embedding algorithms, demonstrating that different training runs produce different results, especially with small underlying datasets. @xcite focuses on the cosine-similarity between words in the learned embedding space, showing large variability upon minor manipulations on the corpus. wendlandt et al. (2018) in this work, we are concerned with the stability of usage-change detection algorithms, and present a metric for measuring this stability. a usagechange detection algorithm takes as input two corpora, and returns a ranked list r of candidate words, sorted from the most likely to have changed to the least likely. for a stable algorithm, we expect different runs to return similar lists. while we do not care about the exact position of a word within a list, we do care about the composition of words at the top of the list. we thus propose a measure we call intersection@k, measuring the percentage of shared words in the the top-k predictions of both outputs: a value of 0 in this measure means that there are no words in the intersection, which indicates high level of variability in the results, while a value of 1 means that all the words are in the intersection, indicating that the results are fully consistent. we expect to see higher intersection@k as k grows. this expectation is confirmed by our experiments in section 7.2. we measure the stability of the usage-change detection algorithms with respect to a change in the underlying word embeddings: we apply the intersection@k metric to two runs of the usagechange detection algorithm on the same corpuspair, where each run is based on a different run of the underlying word embedding algorithm. ## the predominant approach the most prominent method for detecting usage change is that of @xcite , originally applied to detect shifts in dominant word senses across time. it is still the predominant approach in practice, 2 with recent works building upon it @xcite . this method was also shown to be the best performing one among several others @xcite . it works by training word embeddings on the two corpora, aligning the spaces, and then ranking the words by the cosine-distance between their representations in the two spaces, where large distance is expected to indicate significant change in meaning. we refer to this method as aligncos. the alignment is performed by finding an orthogonal linear transformation q that, when given matrices x and y , projects x to y while minimizng the squared loss: this is also indicated by the large number of citations: 350 according to google scholar. the rows of x correspond to embeddings of words in space a, while the rows of y are the corresponding embeddings in space b. this optimization is solved using the orthogonal procrustes (op) method @xcite , that provides a closed form solution. vector space alignment methods are extensively studied also outside of the area of detecting word change, primarily for aligning embedding spaces across language pairs @xcite @xcite . also there, the orthogonal procrustes method is taken to be a top contender @xcite . ## nearest neighbors as a proxy for meaning rather than attempting to project two embedding spaces into a shared space (which may not even map 1:1), we propose to work at the shared vocabulary space. the underlying intuition is that words whose usage changed are likely to be interchangeable with different sets of words, and thus to have different neighbors in the two embedding spaces. this gives rise to a simple and effective algorithm: we represent each word in a corpus as the set of its top k nearest neighbors (nn). we then compute the score for word usage change across corpora by considering the size of the intersection of the two sets (not to be confused with intersection@k defined in section 3): we only consider the words in the intersection of both vocabularies, as words that are rare in one of the corpora are easy to spot using the frequency in the two spaces, and do not neatly fit the definition of usage change. note that our method does not require extensive filtering of words -we only filter words based on their frequency in the corpus foot_1 . we use a large value of @xmath0, because large neighbor sets are more stable than small ones @xcite , leading to improved stability for our algorithm as well. limitations similar to previous methods, our method assumes high quality embeddings, and hence also a relatively large corpus. indeed, in many cases we can expect large quantities of data to be available to the user, especially when considering the fact that the data needed is raw text rather than labeled text. using a limited amount of data results in lower quality embeddings, but also with smaller vocabulary size, which might affect our method. for high-quality embeddings with small vocabulary sizes, we believe that changing k accordingly should suffice. naturally, results will likely degrade as embeddings quality deteriorate. it is also important to note that, like previous approaches, our method does not attempt to provide any guarantees that the detected words have indeed undergone usage change. it is only intended to propose and highlight candidates for such words. these candidates are meant to later be verified by a user who needs to interpret the results in light of their hypothesis and familiarity with the domain. unlike previous methods, as we discuss in section 7.4, our method also provides intuitive means to aid in such an interpretation process. ## experimental setup we compare our proposed method (nn) to the method of @xcite described in section 4 (aligncos), in which the vector spaces are first aligned using the op algorithm, and then words are ranked according to the cosine-distance between the word representation in the two spaces. 5 this method was shown to outperform all others that were compared to it by @xcite . we demonstrate our approach by using it to detect change in word usage in different scenarios. we use the following corpora, whose statistics are listed in we consider three demographics-based distinctions (age, gender, occupation), a day-of-week based distinction, and short-term (4y) diachronic distinctions. we also compare to the longer-term (90y) diachronic setup of @xcite , which is based on google books. author demographics the celebrity profiling corpus @xcite consists of tweets from celebrities along with their traits such as age, gender and occupation. based on these labels, we create the following splits: (1) age: young @xcite @xcite ; (2) gender: male vs. female; (3) occupation: pairwise splits with performer, sports and creator. @xcite collect 580 million tweets in english from june 2009 to february 2010, along with their time-stamps. as this is a fairly large corpus, we consider the tweets of a single month @xcite . we create a split based on the day-of-week: weekday (tweets created on tuesday and wednesday) vs. weekend (tweets created on saturday and sunday). we remove duplicated tweets, as preliminary experiments revealed odd behavior of the representations due to heavily duplicated spam tweets. hebrew diachronic (4y, tweets) the hebrew data we use is taken from a collection of hebrew tweets we collected for several consecutive years, up to 2018. the collection was performed by using the streaming api and filtering for tweets containing at least one of the top 400 most frequent hebrew words. we use the 2014 and 2018 portions of the data, and create a split accordingly. ## related work extensive work has been done on detecting word usage change across corpora that predated the alignment-based methods @xcite @xcite @xcite . in addition, two works are more closely related to our approach. in @xcite , the authors also use the neighbors of a word in order to determine its stability (and therefore, the extent to which it changes). their best model combines the traditional alignment-based approach with weighting the neighbors according to their rank and their stability. the algorithm is iterative, and they update the stability of all the words in the vocabulary in each update step. our method uses the neighbors of the words directly, does not include an iterative process, and does not rely on cosine-distance in the aligned embeddings. in addition, their method requires computation for the whole vocabulary, while other methods, including ours, usually allow querying for a single word. another work that considers the neighbors of the word in order to determine the extent of change is that of @xcite , in which they suggest a measure that is based on the changes of similarities between the target word and its neighbors in both spaces. they find that this method is more suitable for identifying changes that are due to cultural factors, rather than linguistic shift. this may serve as another motivation to move from the global measures to a local one. recent works @xcite explored the possibility of modeling diachronic and usage change using contextualized embeddings extracted from now ubiquitous bert representations @xcite . focusing on the financial domain, montariol and allauzen (2020) use, on top of bert embeddings, a clustering method that does not need to predefine the number of clusters and which leads to interesting results on that domain. another approach from @xcite relies on the inclusion of example-based word sense inventories over time from the oxford dictionary to a bert model. doing so provides an efficient fine-grained word sense representation and enables a seemingly accurate way to monitor word sense change over time. most of those approaches could be easily used with our method, the inclusion of contextualized embeddings would be for example straightforward, we leave it for future work. ## conclusion detecting words that are used differently in different corpora is an important use-case in corpusbased research. we present a simple and effective method for this task, demonstrating its applicability in multiple different settings. we show that the method is considerably more stable than the popular alignment-based method popularized by @xcite , and requires less tuning and word filtering. we suggest researchers to adopt this method, and provide an accompanying software toolkit.