Statistical Uncertainty in Word Embeddings: G lo V e- V (2024)

Abstract: Static word embeddings are ubiquitous in computational social science applications and contribute to practical decision-making in a variety of fields including law and healthcare. However, assessing the statistical uncertainty in downstream conclusions drawn from word embedding statistics has remained challenging. When using only point estimates for embeddings, researchers have no streamlined way of assessing the degree to which their model selection criteria or scientific conclusions are subject to noise due to sparsity in the underlying data used to generate the embeddings. We introduce a method to obtain approximate, easy-to-use, and scalable reconstruction error variance estimates for GloVe, one of the most widely used word embedding models, using an analytical approximation to a multivariate normal model. To demonstrate the value of embeddings with variance (GloVe-V), we illustrate how our approach enables principled hypothesis testing in core word embedding tasks, such as comparing the similarity between different word pairs in vector space, assessing the performance of different models, and analyzing the relative degree of ethnic or gender bias in a corpus using different word lists.

## introduction over the past decade, vector representations of words, or "word embeddings," have become standard ways to quantify word meaning and semantic relationships due to their high performance on natural language tasks @xcite @xcite . word embeddings are now ubiquitous in a wide variety of downstream computational social science applications, including charting the semantic evolution of words over time @xcite , generating simplifications of scientific terminology @xcite , comparing the information density of languages @xcite , assisting in legal interpretation @xcite , and detecting societal biases in educational texts @xcite , historical corpora @xcite , legal documents @xcite , political writing @xcite , and annotator judgments @xcite . task performance metrics using word embeddings also factor prominently into the evaluation of more sophisticated, multimodal artificial intelligence systems, such as brain-computer interfaces @xcite and adversarial text-toimage generation @xcite . though a vast amount of research has relied on a relatively narrow set of word embedding models, no unified framework has emerged for representing statistical uncertainty in how accurately the word embeddings reconstruct the relationships implied by the sample of word co-occurrences. intuitively, we should be less certain about a word's position in vector space the less data we have on its co-occurrences in the raw text @xcite . while it is generally standard practice in the social and natural sciences to check results for statistical significance, the vast majority of applications have relied exclusively on point estimates of embeddings, ignoring uncertainty even when training vectors over smaller corpora (e.g., @xcite . in select applications, approaches have ranged from a bootstrap on the documents in the training corpus @xcite , to permutations of the word list or lexicon @xcite . while useful, such bootstrap and permutation approaches are computationally intractable on large datasets. moreover, they address uncertainty from document or lexicon selection, even though embeddings are parameters of a data generating process on word co-occurrences, not on collections of documents or sets of words @xcite . until now, accounting for the fundamental uncertainty from the data-generating process has eluded the nlp community. to fill this gap, we develop glove-v, a scalable, easy-to-use, computationally efficient method for approximating reconstruction error variances for the glove model @xcite , one of the most widely used word embedding models. our approach leverages the core insight that if context vectors and constant terms are held fixed at optimal values, glove word embeddings are the optimal parameters for a multivariate normal probability model on a weighted log transformation of the rows of the co-occurrence matrix. if we assume that the rows of the co-occurrence matrix are independent given their context vectors and constant terms, the word embedding variances according to this likelihood are computationally tractable on large vocabularies. this assumption is reasonable and is also employed in other settings, such as measuring the influence of particular documents on downstream embedding statistics @xcite . such variance estimates enable researchers to conduct rigorous assessments of model performance and principled statistical hypothesis tests on downstream tasks, responding to the need to account for statistical uncertainty and significance testing in natural language processing and machine learning @xcite @xcite @xcite . our contributions are threefold: (a) we provide the statistical foundations for a principled notion of reconstruction error variances for glove word embeddings; (b) we show that incorporating uncertainty can change conclusions about textual similarities, model selection, and textual bias; (c) we provide a data release including pre-computed word embeddings and variances for the most frequently occurring words in the corpus of american english (coha), the largest corpus of historical american english that is widely used to track the usage and linguistic evolution of english terms over time (e.g., @xcite @xcite @xcite ). 1 1 our code (github.com/reglab/glove-v) and data (nlp.stanford.edu/projects/glove) repositories include data, tutorials, documentation and code to apply the glove-v framework to any downstream task. to make our approach more readily accessible to researchers, we release pre-computed embeddings and variances for coha , intended for academic use. we also plan to make these ## background on glove we use upper case bold letters for matrices x, lower case bold letters for vectors x, and regular non-bolded letters for scalars x, except when indexing into a matrix or vector (i.e., the ij th entry of the matrix x is x ij ). sets are represented by script letters x . word embedding models learn a shared vector space representation of words in a corpus. the training data are word co-occurrences in the corpus, which can be represented by a v ×v co-occurrence matrix x, where x ij is the weighted number of times word j appears in the context of word i, 2 and v is the number of words in the vocabulary. a word embedding is a vector representation of a given word that emerges from the model. the glove word embedding model @xcite learns two embeddings w k , v @xmath0, by minimizing the following cost function for x @xmath1: figure 1 : conceptual diagram of the glove-v method for one word. the top two rows illustrate the structural form and estimation of the original glove model @xcite , which models each row of a logged, weighted co-occurrence matrix as the product of a word vector and context vectors, plus constant terms. as shown in the third row, glove-v creates a distribution for the optimal glove word vector using the reconstruction error found through the glove minimization procedure. these distributions can be efficiently computed word-by-word by assuming conditional independence between words given the optimal context vectors and constants. ## variance derivation for glove-v we now derive the glove-v variance estimator by recasting the optimization problem and recovering a probabilistic interpretation of glove embeddings. ## results to build intuition and demonstrate the usefulness of glove-v variances, we now provide empirical results using the corpus of historical american english (coha) for the 20 th century, which contains english-language texts from a balanced set of genres (fiction, non-fiction, magazines, and newspapers) from 1900-1999 @xcite . 8 we trained multiple glove models for these empirical results. first, we trained a single model on the unaltered coha corpus, which we use for all of the glove-v results. second, we trained 100 glove models on document-level bootstrap samples of the coha (1900-1999) corpus. 9 we trained each 300-dimensional glove model (132m parameters for a vocabulary size of approx- , along with ellipses drawn around 100 draws from the estimated multivariate normal distribution from equation 4 for a random subset of words. lower frequency words like "rigs" and "illumination" have more uncertainty in their estimated positions in the vector space than high frequency words like "she" and "large." the empirical examples we provide in section 5 show how glove-v variances can move researchers in nlp towards emerging best practices for incorporating hypothesis testing in natural language tasks and downstream analyses @xcite . ## glove-v enables principled significance testing we now show how glove-v enables statistical significance testing, addressing the increasing recognition for nlp to move beyond point estimates alone @xcite . glove-v can also help researchers assess when a corpus is underpowered for specific inferences, as we illustrate below. ## conclusion in this paper, we have derived, computed, and demonstrated the utility of glove-v, a new method to represent uncertainty in word embedding locations using the seminal glove model. the glove-v variances provide researchers with the ability to easily propagate uncertainty in the word embeddings to downstream test statistics of interest, such as word similarity metrics that are used in both evaluation and analyses involving word embeddings. unlike methods such as the document and word list bootstrap, the method is computationally efficient even on large corpora and represents uncertainty due to sparsity in the underlying co-occurrence matrix, which is often invisible in downstream analyses that use only the estimated word embeddings. as we have shown in section 5, incorporating uncertainty into downstream analyses can have consequential impacts on the conclusions researchers draw, and should be a best practice moving forward for studies that use word embeddings to infer semantic meaning. finally, while outside the scope of the current study, we note that the contextual word and passage representations of transformer large language models are also point estimates and similar questions of embedding uncertainty apply when using such models as well.