COIN – an Inexpensive and Strong Baseline for Predicting Out of Vocabulary Word Embeddings (2022)

Abstract: Social media is the ultimate challenge for many natural language processing tools. The constant emergence of linguistic constructs challenge even the most sophisticated NLP tools. Predicting word embeddings for out of vocabulary words is one of those challenges. Word embedding models only include terms that occur a sufficient number of times in their training corpora. Word embedding vector models are unable to directly provide any useful information about a word not in their vocabularies. We propose a fast method for predicting vectors for out of vocabulary terms that makes use of the surrounding terms of the unknown term and the hidden context layer of the word2vec model. We propose this method as a strong baseline in the sense that 1) while it does not surpass all state-of-the-art methods, it surpasses several techniques for vector prediction on benchmark tasks, 2) even when it underperforms, the margin is very small retaining competitive performance in downstream tasks, and 3) it is inexpensive to compute, requiring no additional training stage. We also show that our technique can be incorporated into existing methods to achieve a new state-of-the-art on the word vector prediction problem.

## introduction in recent years, distributive models of lexical semantics, i.e., word embedding models, have proven to be a very useful tool for representing natural language terms (i.e., words and common phrases such as 'new york') as real numbered vectors. these models, such as word2vec @xcite , glove @xcite , and fasttext @xcite , use a large corpus of documents to learn an embedded vector representation of lexical units based on their co-occurrence within sentences. word embedding vectors capture semantic features of the terms, for example synonymous terms will be nearby in the embedded vector space. this makes word embeddings more powerful than other lexical representations such as one-hot vector encodings. furthermore, distributive models are derived using unsupervised algorithms, which are efficient to run on large corpora. a major limitation, however, is that to derive a high quality word embedding vector for a term, it must occur more than a certain times in the training corpus @xcite . terms that do not meet this threshold are not included in the model's vocabulary. when using a word embedding model, a user may encounter a large number of out of vocabulary (oov) terms for various reasons. when we encounter oov terms in a document, a common strategy is to simply ignore the unknown term. alternately we can represent the term by a zero vector, or by the average of all known word vectors, but these strategies clearly do not extract any meaningful information from the oov item. a number of recent efforts propose techniques for predicting high quality word embedding vectors for oov terms from the information available on the term in situ @xcite @xcite @xcite @xcite ). an important downside of these methods in practice is that they take an unpredictable amount of time to develop and deploy. a question arise in this context for deployment in an nlp pipeline: can one employ (and deploy) a simpler solution (with minimal accuracy loss) until one creates a more advanced solution? the goal of this work is to seek answers to this question in the context of oov terms. in this work, we present a context-based method to predict the word embedding vectors for oov terms. our method makes use of context information that is learned internally in the course of training a word embedding model such as word2vec. we call this the coin (context information) method. we advocate coin as a competitive baseline in the following sense: accurate (compared to state-of-the-art models). the performance of coin is close, both in direct comparison and downstream tasks, to state-of-the-art models. we compare coin with four such models (add, n2v, alc, and fcm) on the benchmark tasks defnonce and crw; coin vectors meet or exceed the accuracy of all except for fcm. inexpensive. generating a coin vector prediction requires only a pretrained word2vec model and runs on a regular personal laptop (without the requirement of gpu). the only computational cost of coin is the equivalent of a few look-ups of wordvector mappings. it is able to generate high quality predicted word vectors for oov terms from as few as one observed occurrence without any additional training. while the state-of-the-art method fcm requires to train a set of word vectors and then in a second step trains a model for oov predictions. fast. coin is much faster than other state-ofthe-art models. it gives users immediate feedback (in seconds). while most supervised techniques for word embedding for oov terms require hours of computation before a user can analyze it. in our experiments, the coin method takes 131 seconds to generate the vectors of oov terms for evaluation on the defnonce task, and it only takes 21 seconds to run all of the 2, 4, and 6 sentence evaluations on the chim task. in comparison, fcm takes nearly 7 hours of training time on the defnonce task. balancing gains with cost. oov words are important in downstream tasks @xcite @xcite @xcite and not as a standalone exercise. we report a study on 7 senteval tasks, getting a .002 prediction difference between coin at 73.9% and fcm at 74.1%. with the compute and energy demands of many modern nlp methods growing exponentially, one needs to consider weighing energy costs with the performance gains of a new model before deploying it @xcite . in our setting, the gain of energy hungry methods is marginal compared to coin. however, many of the alternative approaches require a second step than training the initial word embedding model. all of the training required for coin is accomplished with the original word embedding step as a holistic model minimizing total parameters, improved performance and a better model design. we can draw a parallel from daily life. toyota corolla is an excellent means of transportation for millions of people. ferrari is another excellent means of transportation, but only available to the few who can afford it. one may view coin (and methods in its class) as the "corolla" among the methods of predicting embeddings for oovs and accept that its performance may fall short of the "ferrari's" (e.g., fasttext, elmo and bert), one also needs to accept that the "ferrari's" requires more resources: fuel (i.e., energy) and maintenance (i.e., hardware). often scientists and engineers need a "corolla" to deliver a proof of concept. there are times when their hardware infrastructures are illsuited to run the "ferrari's," e.g., lacking tens of gpus. our goal in this paper is to argue that there is a place for "corolla's," even though we all seek shiny "ferrari's." coin vectors are suitable on their own for word vector prediction: they can be used to initialize a trained method. we show that using coin vectors as the first step in existing vector predicting systems can set a new state-of-the-art on benchmark tasks. we also notice that coin vectors may reduce the training cost for those methods, improving the accuracy of the component techniques. we show these on fcm in section 5.5. in summary, the contributions of this paper are: ## word embeddings word embeddings are a technique for learning a vector representation of terms from a corpus based on their distributional features, that is, the contexts within which a term is observed @xcite @xcite . word embedding techniques belong among a class of machine learning algorithms where the model is trained to minimize error on a dummy task and the true purpose of training is to extract an intermediate, internal representation of the input data. in the word embedding case, the dummy task is that of predicting the surrounding terms of a target term (or vice versa) and the true desired output is the internal representation of the input terms, i.e., the word embedding vectors. the various methods for learning word embeddings ontext govan_word n_word _context g_word ontext mayday_word yday_context t word text 15 1 bavarian_word bavarian_context kenya_word kenya_context govan_word govan_context evesham_word evesham_context luton_word luton_context pudding_word pudding_context mayday_word mayday_context brine_word brine_context sunglasses_word sunglasses_context patchwork_word patchwork_context @xmath0. target similarity: 0.37 --> 0.89 reference similarity: -0.17 --> 0.15 figure 1: a t-sne visualization of the relative locations of word and context vectors for a sample vocabulary. the word and context spaces are aligned using absolute orientation with translation and scaling @xcite . for most terms the word and context vectors are respectively well separated. word2vec is one of the most popular technique to learn word embeddings @xcite . in word2vec model, the word vectors are not always close to the context vectors. in figure 1 we show a visualization of the word and context vectors for some terms from the vocabulary. we find that the context and word vectors are quite far from each other for most terms. this general difference between vectors demonstrates that summing the context vectors will lead to a very different result from summing over the word vectors. this suggests that context vectors encapsulate information that is not captured by the word vectors, which may be useful to predict embeddding vectors for oov terms. we aim to show this in this work. we note here that figure 1 was generated after the word and context vector were aligned. we align the word and context vectors using absolute orientation with translation and scaling @xcite . other orientation algorithms are available, but this gave the best average cosine similarity between the vectors in our experiments. we also note that the context and word vectors as plotted in figure 1 are even farther apart before alignment was performed. following our study of word embeddings, we observe that the word embedding vector w i of a word is trained to be similar to the context vectors of the words in its context window, c @xmath1, rather than the word vectors, which have previously been used in the prediction task. intuition suggests that by using the context vectors of the context words we can improve the accuracy of our predicted vectors. ## related work previous techniques to predict embedding vectors for oov terms usually follow two theoretical lines: 1) using morphological or form features of the oov term, and 2) extracting information from the surrounding context terms. form based methods in category 1) use some characteristics of the representation of the term, such as letter combinations. @xcite use morphological features. @xcite develops the fasttext model, which is similar to word2vec in design, which operates on character n-grams rather than whole words. in some tasks, however, an oov term may not have any character representation, or it may have an arbitrary representation, for example, the oov term may be represented by underscores: "___" @xcite or by a made-up word form @xcite . in the downstream task there may be no available form for the term, such as "filling in the missing word." for the context based works in category 2), @xcite develop the additive method, which takes the sum of the word vectors of the surrounding terms. the approach of @xcite initializes each oov term as the sum of the word vectors of the known terms in the context sentences, downsampling frequent words, and then runs a modified word2vec training procedure on the oov terms only, with a highly accelerated learning rate. @xcite finds a linear transformation a and uses it to transform the sum of the context word vectors to the word vector of oov terms. recently @xcite propose a hybrid approach using both form and context. it learns a neural network model, which encodes information about the form (n-grams) and likelihood the form will contribute to an understanding of the meaning of the term, as well as a linear transformation similar to @xcite . @xcite ) further proposes a method to identify high quality contexts for inferring the oov term vector. coin vectors squarely fall into 2), context based methods. our key observation is that there exists an additional set of vectors, one for each in-vocabulary item, which are trained internally by the word2vec model. in word2vec, each term is represented by two latent vectors: a word vector and a context vector. these vectors are better suited, by design, for deducing oov term vectors from the surrounding context words. when the model is being trained, these additional context vectors are used to represent the terms when they are found in the context window of the target term. these vectors can be easily extracted from a word2vec model. recently contextualized word vectors (cwv) methods such as elmo and bert have received much attention in the nlp community @xcite @xcite . cwv prediction shares some similarity with oov vector prediction, but they are intrinsically distinct problems. cwvs derive a vector representation for a term based on the surrounding context in each usage, thus there is not a generalized concept of a word vector for a lexical item as is the case with traditional word vectors. while in traditional word vector models (like word2vec), each term gets one vector, which is used regardless of the surrounding terms. we include them in our empirical study on the oov benchmark tasks. ## methodology based on our observation discussed previously, we suggest that the context vectors of the context words can be used to predict the word embedding vectors for oov terms. we introduce our method, coin, which uses context vectors only to solve the prediction problem for oov terms. ## evaluation in this section, we compare our method coin to a number of state-of-the-art models to show the efficiency and effectiveness of coin. we also show that coin vectors can be used in conjunction with an existing, trained method to yield improved results over either method alone ## conclusion in this work we present the coin method for predicting word embedding vectors for oov terms using the context vectors of context words. we advertise coin as an inexpensive and strong baseline. we show that coin performs close to the existing state-of-the-art techniques, while being much faster as it requires no additional training. we also show how coin can be used along with existing techniques to give a new state-of-the-art on vector prediction tasks, and how it can be used to help downstream tasks such as sentiment analysis. besides, we explore different model settings for learning the word vectors. the results of coin are robust across model choices, with generally better performance from cbow trained embeddings.