Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings (2022)

Abstract: Although contextualized embeddings generated from large-scale pre-trained models perform well in many tasks, traditional static embeddings (e.g., Skip-gram, Word2Vec) still play an important role in low-resource and lightweight settings due to their low computational cost, ease of deployment, and stability. In this paper, we aim to improve word embeddings by 1) incorporating more contextual information from existing pre-trained models into the Skip-gram framework, which we call Context-to-Vec; 2) proposing a post-processing retrofitting method for static embeddings independent of training by employing priori synonym knowledge and weighted vector distribution. Through extrinsic and intrinsic tasks, our methods are well proven to outperform the baselines by a large margin.

## introduction contextualized embeddings such as bert @xcite and roberta @xcite have become the default architectures for most downstream nlp tasks. however, they are computationally expensive, resource-demanding, hence environmentally unfriendly. compared with contextualized embeddings, static embeddings like skipgram @xcite and glove @xcite are lighter and less computationally expensive. furthermore, they can even perform without significant performance loss for contextindependent tasks like lexical-semantic tasks (e.g., word analogy), or some tasks with plentiful labeled data and simple language @xcite . recent work has attempted to enhance static word embedding while maintaining the benefits of both contextualized embedding and static embedding. among these efforts, one category is the direct conversion of contextualized embeddings in the context-to-vec phase, static word embeddings are trained using contextualized embeddings based on a monolingual corpus. while in the post-processing phase, external knowledge is introduced to fine-tune the word vectors based on the graph topology. to static embeddings @xcite . the other category of enhancement is to make use of contextualized embeddings for static embeddings @xcite . the latter category is a newer paradigm, which we call context-to-vec. this paradigm not only alleviates the word sense ambiguities from static embedding, but also fuses more syntactic and semantic information in the context within a fixed window. for the context-to-vec paradigm, an association between contextualized word vectors and static word vectors is essentially required. in this case, the contextualized signal serves as a source of information enhancement for the static embeddings @xcite . however, the existing efforts only consider the contextualized embeddings of center words as the source, which is actually incomplete since the contextualized features for the context words of the center words are ignored. in addition, benefiting from the invariance and stability of already trained static embeddings, postprocessing for retrofitting word vectors is also an effective paradigm for improving static embeddings. for example, one solution is an unsupervised approach that performs a singular value decomposition to reassign feature weights @xcite , but this does not utilize more external knowledge and lacks interpretation. poor initial spatial distribution of word embeddings obtained from training may lead to worse results. another common solution is to use a synonym lexicon @xcite , which exploits external prior knowledge with more interpretability but does not take into account the extent of spatial distance in the context. in this work, we unify the two paradigms above within a model to enhance static embeddings. on the one hand, we follow the context-to-vec paradigm in using contextualized representations of center words and their context words as references for static embeddings. on the other hand, we propose a graph-based semi-supervised postprocessing method by using a synonym lexicon as prior knowledge, which can leverage proximal word clustering signals and incorporate distribution probabilities. the overall training pipeline is shown in fig. 1 . the pipeline is divided into two separate phases, where the first phase follows the context-to-vec paradigm by distilling contextualized information into static embeddings, while the second phase fine-tunes the word embeddings based on graph topology. to validate our proposed methods, we evaluate several intrinsic and extrinsic tasks on public benchmarks. the experimental results demonstrate that our models significantly outperform traditional word embeddings and other distilled word vectors in word similarity, word analogy, and word concept categorization tasks. besides, our models moderately outperform baselines in all downstream clustering tasks. to our knowledge, we are the first to train static word vectors by using more contextual knowledge in both training and post-processing phases. the code and trained embeddings are made available at https://github.com/ binbinjiang/context2vector . context-to-vec. the fusion of contextualized and static embeddings is a newly emerged paradigm in recent years. for instance, @xcite propose syngcn using gcn to calculate context word embeddings based on syntax structures; bommasani et al. (2020) introduce a static version of bert embeddings to represent static embeddings; wang et al. (2021) enhance the skip-gram model by distilling contextual information from bert. our work also follows this paradigm but introduce more context constraints. post-processing embeddings. post-processing has been used for improving trained word embeddings. typically, faruqui et al. (2014) use synonym lexicons to constrain the semantic range; artetxe @xcite propose a method based on eigenvalue singular decomposition. similar to these techniques, our post-processing method is easy for deployment and can be applied to any static embeddings. the difference is that we not only take advantage of the additional knowledge, but also consider the distance weights of the word vectors, overcoming the limitations of existing methods with better interpretability. ## experiments we use wikipedia to train static embeddings. the cleaned corpus has about 57 million sentences and 1.1 billion words. the total number of vocabularies is 150k. sentences between 10 and 40 in length were selected during training. ## conclusion we considered improving word embeddings by integrating more contextual information from existing pre-trained models into the skip-gram framework. in addition, based on inherent properties of static embeddings, we proposed a graph-based postretrofitting method by employing priori synonym knowledge and a weighted distribution probability. the experimental results show the superiority of our proposed methods, which gives the best results on a range of intrinsic and extrinsic tasks compared to baselines. in future work, we will consider prior knowledge directly during training to avoid a multistage process.