Obtaining Better Static Word Embeddings Using Contextual Embedding Models (2021)

Abstract: The advent of contextual word embeddings — representations of words which incorporate semantic and syntactic information from their context—has led to tremendous improvements on a wide variety of NLP tasks. However, recent contextual models have prohibitively high computational cost in many use-cases and are often hard to interpret. In this work, we demonstrate that our proposed distillation method, which is a simple extension of CBOW-based training, allows to significantly improve computational efficiency of NLP applications, while outperforming the quality of existing static embeddings trained from scratch as well as those distilled from previously proposed methods. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings via standard lexical evaluation tasks.

## introduction word embeddings-representations of words which reflect semantic and syntactic information carried by them are ubiquitous in natural language processing. static word representation models such as glove @xcite , cbow, skipgram @xcite and sent2vec @xcite obtain stand-alone representations which do not depend on their surrounding words or sentences (context). contextual embedding models @xcite @xcite @xcite on the other hand, embed the contextual information as well into the word representations making them more expressive than static word representations in most use-cases. while recent progress on contextual embeddings has been tremendously impactful, static embeddings still remain fundamentally important in many scenarios as well: we also refer the reader to this article 2 illustrating several down-sides of using bert-like models over static embedding models for non-specialist users. indeed, we can see continued prevalence of static word embeddings in industry and research areas including but not limited to medicine @xcite @xcite and social sciences @xcite @xcite . from a cognitive science point of view, human language has been hypothesized to have both con-1 bert base @xcite produces 768 dimensional word embeddings using 109m parameters, requiring 29b flops per inference call @xcite ). 2 do humanists need bert? ( https:// tedunderwood.com/2019/07/15/ ) most existing word embedding models, whether static or contextual, follow @xcite 's famous hypothesis -"you shall know a word by the company it keeps" , i.e., the meaning of a word arises from its context. during training existing static word embedding models, representations of contexts are generally approximated using averaging or sum of the constituent word embeddings, which disregards the relative word ordering as well as the interplay of information beyond simple pairs of words, thus losing most contextual information. ad-hoc remedies attempt to capture longer contextual information per word using higher order n-grams like bigrams or trigrams, and have been shown to improve the performance of static word embedding models @xcite . however, these methods are not scalable to cover longer contexts. in this work, we obtain improved static word embeddings by leveraging recent contextual embedding advances, namely by distilling existing contextual embeddings into static ones. our proposed distillation procedure is inspired by existing cbow-based static word embedding algorithms, but during training plugs in any existing contextual representation to serve as the context element of each word. our resulting embeddings outperform the current static embedding methods, as well as the current state-of-the-art static embedding distillation method on both unsupervised lexical similarity tasks as well as on downstream supervised tasks, by a significant margin. the resulting static embeddings remain compatible with the underlying contextual model used, and thus allow us to gauge the extent of lexical information carried by static vs contextual word embeddings. we release our code and trained embeddings publicly on github foot_0 . ## related work a few methods for distilling static embeddings have already been proposed. @xcite propose using contextual embeddings of the same word in a large number of different contexts. they take the first principal component of the matrix formed by using these embeddings as rows and use it as a static embedding. however, this method is not scalable in terms of memory (the embedding matrix scaling with the number of contexts) and computational cost (pca). @xcite propose two different approaches to obtain static embeddings from contextual models. other related work includes distillation of contextual word embeddings to obtain sentence embeddings @xcite . we also refer the reader to @xcite for a discussion on the semantic properties of contextual models (primarily bert) as well as @xcite , a survey on different works exploring the inner workings of bert including its semantic properties. ## proposed method to distill existing contextual word representation models into static word embeddings, we augment a cbow-inspired static word-embedding method as our anchor method to accommodate additional contextual information of the (contextual) teacher model. sent2vec @xcite is a modification of the cbow static word-embedding method which instead of a fixed-size context window uses the entire sentence to predict the masked word. it also has the ability to learn n-gram representations along with unigram representations, allowing to better disentangle local contextual information from the static unigram embeddings. sent2vec, originally meant to obtain sentence embeddings and later repurposed to obtain word representations @xcite was shown to outperform competing methods including glove @xcite , cbow, skipgram @xcite and fasttext @xcite on word similarity evaluations. for a raw text corpus c (collection of sentences), the training objective is given by here, w t is the masked target word, u and v are the target word embedding and the source n-gram matrices respectively, n is the set of negative target samples and, ℓ : @xmath0. for sent2vec, the context encoder e ctx used in optimizing (1) is simply given by the (static, non-contextual) sum of all vectors in the sentence without the target word, we will now generalize the objective (1) by allowing the use of arbitrary modern contextual representations e ctx instead of the static context representation as in (2). this key element will allow us to translate quality gains from improved contextual representations also to better static word embedding in the resulting matrix u . we propose two different approaches of doing so, which differ in the granularity of context used for obtaining the contextual embeddings. ## conclusion and future work this work proposes to augment earlier word2vec-based methods by leveraging recent more expressive deep contextual embedding models to extract static word embeddings. the resulting distilled static embeddings, on an average, outperform their competitors on both unsupervised as well downstream supervised evaluations and thus can be used to replace compute-heavy contextual embedding models (or existing static embedding models) at inference time in many compute-resource-limited applications. the resulting embeddings can also be used as a task-agnostic tool to measure the lexical information conveyed by contextual embedding models and allow a fair comparison with their static analogues. further work can explore extending this distillation framework into cross-lingual domains @xcite as well as using better pooling methods instead of simple averaging for obtaining the context representation, or joint fine-tuning to obtain even stronger static word embeddings. another promising avenue is the use of a similar approach to learn sense embeddings from contextual embedding models. we would also like to investigate the performance of these embeddings when distilled on a larger corpus along with more extensive hyperparameter tuning. last but not the least, we would like to release x2static models for different languages for further public use.