Estimating word co-occurrence probabilities from pretrained static embeddings using a log-bilinear model (2022)

Abstract: We investigate how to use pretrained static word embeddings to deliver improved estimates of bilexical co-occurrence probabilities: conditional probabilities of one word given a single other word in a specific relationship. Such probabilities play important roles in psycholinguistics, corpus linguistics, and usage-based cognitive modeling of language more generally. We propose a log-bilinear model taking pretrained vector representations of the two words as input, enabling generalization based on the distributional information contained in both vectors. We show that this model outperforms baselines in estimating probabilities of adjectives given nouns that they attributively modify, and probabilities of nominal direct objects given their head verbs, given limited training data in Arabic, English, Korean, and Spanish.

## introduction word co-occurrence probabilities are a key ingredient in usage-based cognitive models of language. by word co-occurrence probabilities, i mean the probability of a word w given some other single word c, p(w | c), where words w and c have some specific relationship, for example adjectives that attributively modify nouns or nouns serving as direct objects of verbs @xcite . these co-occurrence probabilities are psycholinguistically relevant because they feed into information-theoretic measures of 'thematic fit' and selectional restriction @xcite @xcite which are relevant in predicting human online processing difficulty (e.g. @xcite , and play a key role in language acquisition @xcite . most prominently, the widely-used pointwise mutual information (pmi) measure of association strength, pmi (w, c) = log p(w|c) p(w) @xcite , relies on these condi-tional probabilities as an input. pmi makes appearances in models of grammar induction from text @xcite @xcite , online sentence comprehension and production @xcite , and quantitative theories of word order variation @xcite . word co-occurrence probabilities are hard to estimate accurately from text data because empirical counts of a particular pair of words in a particular relation are often sparse. this limitation makes it hard to evaluate cognitive theories that operate on co-occurrence probabilities. although high-performance pretrained language models now exist @xcite , the probabilities of interest often cannot be read off of these models directly, because w and c might be defined by relations that cannot be straightforwardly detected in terms of linear word order or templates. for example, suppose we are interested in the distribution of adjectives attributively modifying a noun in english. it would not do to ask a language model for the distribution of words immediately preceding a noun, because some of these words will not be attributive adjectives. i propose to improve the estimation of word cooccurrence probabilities by leveraging pretrained static word embeddings to enhance generalization from potentially small training sets. my method enables generalization based on the semantic and syntactic information contained in word embeddings for both words w and c. ## model setting we are given a vocabulary of words v , a finite target word set w ⊆ v , a dataset of n pairs of words { w i , c i } n @xmath0, and a pretrained mapping from words to d-dimensional static embeddings e : @xmath1. supposing the dataset consists of iid samples from some distribution p (w, c) = p(c)p(w | c), our goal is to find a conditional distribution q(w | c) with support w to approximate p(w | c) in a way that leverages the static embeddings e. proposed model i propose a log-bilinear model @xcite using word embeddings as input: modeling decisions a modeler applying this approach needs to make a number of decisions, including the choice of static word embeddings and the structure of the word encoders φ(•) and ψ(•). it is also possible to set ψ = φ, using the same function to encode both the target word and the context word; this setup can reduce the number of parameters at the cost of less flexibility in fitting the training data. another major modeling decision involves the target word vocabulary w , which determines the support of q(w | c) and is summed over during the calculation of the partition function (eq. 2). in some cases, the modeler may not have access to a finite set w of possible target words. as long as the full vocabulary v is finite, it is possible to set @xmath2. setting @xmath3, thus avoiding the risk of prematurely excluding legitimate target words. it has the 1 i have suppressed bias terms from the notation. ## related work distributional similarity information has been used to improve modeling of word co-occurrence probabilities in previous work. @xcite defined a kernel-based interpolated language model where probability mass is explicitly spread over similar words, with variant models along these lines found in @xcite and yarlett @xcite . these models leverage similarity information about target words but not context words. in contrast, @xcite proposed a method which uses similarity information about the context word but not the target word. @xcite developed a method that can exploit similarity information about both target and context, using a markov chain algorithm incorporating distributional and wordnet similarities. none of this previous work derived word similarity information from pretrained embeddings, because such embeddings did not exist at the time. the log-bilinear model for conditional word probabilities was introduced in a language modeling context by @xcite . @xcite influentially proposed to use the vector representations output by the word encoder in such a model as general word embeddings. the current work aims to return log-bilinear models to their language modeling roots, evaluating the capabilities of these models to estimate co-occurrence probabilities using pretrained embeddings as input, with a focus on word distributions where training data is limited. here the target word vocabulary is typically small enough that the partition function (eq. 2) can be computed directly on modern hardware, so that approximations such as noisecontrastive estimation @xcite are not necessary. recently @xcite introduced a neural-bayesian nonparametric estimator for probability distributions on single words. their setting has an unknown and generally infinite vocabulary v , and their model generalizes using a characterlevel lstm. in contrast, the current model assumes a pre-existing known vocabulary v with embeddings, and generalizes based on those embeddings. a hybrid model may be possible in future work. a related literature in corpus linguistics and nlp has explored the nature of restricted binary word co-occurrences, called collocations (for recent examples, see @xcite @xcite . this work focuses narrowly on the estimation of bilexical conditional probabilities, which are often inputs to models for collocation detection. ## experiments i study the ability of the embedding-based logbilinear model to estimate conditional distributions for (1) adjectives attributively modifying nouns and (2) nominal direct objects modifying verbs, in arabic, english, korean, and spanish. i compare the model against baselines: all baselines are subject to the same vocabulary restrictions and out-of-vocabulary policy as the full log-bilinear models. as a standard test metric, i report the average negative log likelihood (nll) of held-out data. i report nlls for the full test set, as well as the challenging subset of the test set consisting of word pairs where the context word was never seen during training. below, i describe the experimental setting for the two tasks, and then i describe the results. ## conclusion i evaluated log-bilinear modeling as means to leverage pretrained word embeddings for the es-57 timation of co-occurrence probabilities in different syntactic configurations. i found that this method delivers accurate probability estimates across languages, outperforming baselines. this method will be useful in all applications requiring such probabilities. code implementing the method can be found at https://github. com/langprocgroup/vectorprob .