Analyzing the Surprising Variability in Word Embedding Stability Across Languages (2021)

Abstract: Word embeddings are powerful representations that form the foundation of many natural language processing architectures, both in English and in other languages. To gain further insight into word embeddings, we explore their stability (e.g., overlap between the nearest neighbors of a word in different embedding spaces) in diverse languages. We discuss linguistic properties that are related to stability, drawing out insights about correlations with affixing, language gender systems, and other features. This has implications for embedding use, particularly in research that uses them to study language trends.

## introduction word embeddings have become an established part of natural language processing (nlp) @xcite . stability, defined as the overlap between the nearest neighbors of a word in different embedding spaces, was introduced to measure variations in local embedding neighborhoods across changes in data, algorithms, and word properties @xcite . these studies found that many common english embedding spaces are surprisingly unstable, which has implications for work that uses embeddings as features in downstream tasks, and work that uses embeddings to study specific properties of language. however, research to date on word embedding stability has been exclusively done on english and so is not representative of all languages. in this work, we explore the stability of word embeddings in a wide range of languages. better understanding the differences caused by diverse languages will provide a foundation for building embeddings and nlp tools in all languages. 1 in english and other very high resource languages, it has become common practice to use contextualized word embeddings, such as bert @xcite and xlnet @xcite . these algorithms require huge amounts of computational resources and data. for example, it takes 2.5 days to train xlnet with 512 tpu v3 chips. in addition to requiring heavy computational resources, most contextualized embedding algorithms need large amounts of data. bert uses 3.3 billion words of training data. in contrast to these large corpora, many datasets from low-resource languages are fairly small @xcite . to support scenarios where using huge amounts of data and computational resources is not feasible, it is important to continue developing our understanding of context-independent word embeddings, such as word2vec @xcite and glove @xcite . these algorithms continue to be used in a wide variety of situations, including the computational humanities @xcite and languages where only small corpora are available @xcite . in this work, we consider how stability varies for different languages, and how linguistic properties are related to stability-a previously understudied relationship. using regression modeling, we capture relationships between linguistic properties and average stability of a language, and we draw out insights about how linguistic features relate to stability. for instance, we find that embeddings in languages with more affixing tend to be less stable. our findings provide crucial context for research that uses word embeddings to study language properties and trends (e.g., @xcite , which often rely on raw embeddings created by glove or word2vec. if these embeddings are unstable, then research using them needs to take this into account in terms of methodologies and error analysis. ## related work word embeddings are low-dimensional vectors used to represent words, normally in downstream tasks, such as word sense disambiguation @xcite ) and text summarization @xcite . they have been shown to capture both syntactic and semantic properties of words, making them useful in a wide range of nlp tasks @xcite . in this work, we explore word embeddings that generate one embedding per word, regardless of the word's context. we consider two widely used algorithms: word2vec @xcite and glove @xcite . our work analyzes embeddings in multiple languages, which is important because embeddings are commonly used across many languages. in particular, there has been interest in embeddings for low-resource languages @xcite . in this work, we use stability to measure the quality of word embeddings. similar to the work we present here on stability, other research looks at how nearest neighbors vary as properties of the embedding spaces change. @xcite found that the lowest frequency and the highest frequency words have the highest variation among nearest neighbors. additional research has explored how semantic and syntactic properties of words change with different embedding algorithm and parameter choices @xcite . unlike our work, previous studies only considered english. finally, while our work is not a form of embedding evaluation, it is related to the topic @xcite @xcite . there has been extensive work on evaluating word embeddings, seen in the recent repeval workshops @xcite , and going back to work comparing them with counting based methods @xcite . our findings indicate that work on embedding evaluation should take into consideration stability, using multiple training runs to confirm results. similarly, stability should be considered when studying the impact of embeddings on downstream tasks. @xcite specifically looked at the downstream instability of word embeddings, and found that there is a stability-memory tradeoff, and higher stability can be achieved by increasing the embedding dimension. ## data in order to explore the stability of word embeddings in different languages, we work with two datasets, wikipedia and the bible. while wikipedia has more data, the bible covers more languages. wikipedia is a comparable corpus, whereas the bible is a parallel corpus. ## calculating stability in many languages the first part of our work is a comparison of stability across languages. before presenting our measurements, we define stability and analyze some important methodological decisions. model 1: indie, punk, progressive, pop, roll, band, blues, brass, class, alternative model 2: punk, indie, alternative, progressive, band, sedimentary, bands, psychedelic, climbing, pop model 3: punk, pop, indie, alternative, band, roll, progressive, folk, climbing, metal ## regression modeling we now explore linguistic factors that correlate with stability. to draw conclusions about specific linguistic features, we use a ridge regression model @xcite 10 to predict the average stability of all words in a language given features reflecting language properties. regression models have previously been used to measure the impact of individual features @xcite . ridge regression regularizes the magnitude of the model weights, producing a more interpretable model than non-regularized linear regression. we experiment with different regularization strengths and use the best-performing value (α = 10). 11 we choose to use a linear model here because of its interpretability. while more complicated models might yield additional insight, we show that there are interesting connections to be drawn from a linear model. ## results and discussion suffixes and prefixes. for all three of these properties, more affixing is associated with lower stability. when considering word embeddings, this result makes intuitive sense. affixes cause there to be many different word variations (e.g., walk, walked, walking, walker), which may not be handled consistently by the embedding algorithm, leading to lower average stability. ## conclusion in this paper, we considered how stability varies across different languages. this work is important because algorithms such as glove and word2vec continue to be effective methods in a wide variety of scenarios @xcite , particularly the computational humanities and languages where large corpora are not available. we studied the relationship between linguistic properties and stability, something that has been previously understudied. we drew out several aspects of this relationship, including that languages with more affixing tend to have less stable embeddings, and languages with no gender systems tend to have more stable embeddings. these insights can be used in future work to inform the design of embeddings in many languages. for example, this work suggests that future embedding space designs need to take into account gendered words and morphologically rich words with affixes.