Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection (2021)

Abstract: One of the long-standing challenges in lexical semantics consists in learning representations of words which reflect their semantic properties. The remarkable success of word embeddings for this purpose suggests that high-quality representations can be obtained by summarizing the sentence contexts of word mentions. In this paper, we propose a method for learning word representations that follows this basic strategy, but differs from standard word embeddings in two important ways. First, we take advantage of contextualized language models (CLMs) rather than bags of word vectors to encode contexts. Second, rather than learning a word vector directly, we use a topic model to partition the contexts in which words appear, and then learn different topic-specific vectors for each word. Finally, we use a task-specific supervision signal to make a soft selection of the resulting vectors. We show that this simple strategy leads to high-quality word vectors, which are more predictive of semantic properties than word embeddings and existing CLM-based strategies.

## introduction in the last few years, contextualized language models (clms) such as bert @xcite have largely replaced the use of static (i.e. noncontextualized) word vectors in many natural language processing (nlp) tasks. however, static word vectors remain important in applications where word meaning has to be modelled in the absence of (sentence) context. for instance, static word vectors are needed for zero-shot image classification @xcite and zero-shot entity typing @xcite , for ontology alignment @xcite and completion @xcite , taxonomy learning @xcite , or for representing query terms in information retrieval systems @xcite . moreover, @xcite recently found that static word vectors can complement clms, by serving as anchors for contextualized vectors, while @xcite found that incorporating static word vectors could improve the performance of bert for social media classification. given the impressive performance of clms across many nlp tasks, a natural question is whether such models can be used to learn highquality static word vectors, and whether the resulting vectors have any advantages compared to those from standard word embedding models @xcite . a number of recent works have begun to explore this question @xcite @xcite . broadly speaking, the idea is to construct a static word vector for a word w by randomly selecting sentences in which this word occurs, and then averaging the contextualized representations of w across these sentences. since it is not usually computationally feasible to run the clm on all sentences mentioning w, a sample of such sentences has to be selected. this begs the question: how should these sentences be chosen? in the aforementioned works, sentences are selected at random, but this may not be optimal. if we want to use the resulting word vectors in downstream tasks such as zero-shot learning or ontology completion, we need vectors that capture the salient semantic properties of words. intuitively, we should thus favor sentences that best reflect these properties. for instance, many of the mentions of the word banana on wikipedia are about the cultivation and export of bananas, and about the specifics of particular banana cultivars. by learning a static word vector from such sentences, we may end up with a vector that does not reflect our commonsense understanding of bananas, e.g. the fact that they are curved, yellow and sweet. the main aim of this paper is to analyze to what extent topic models such as latent dirichlet allocation @xcite can be used to address this issue. continuing the previous example, we may find that the word banana occurs in wikipedia articles on the following topics: economics, biology, food or popular culture. while most mentions might be in articles on economics and biology, it is the latter two topics that are most relevant for modelling the commonsense properties of bananas. note that the optimal selection of topics is taskdependent, e.g. in an nlp system for analyzing financial news, the economics topic would clearly be more relevant. for this reason, we propose to learn a word vector for each topic separately. since the optimal choice of topics is task-dependent, we then rely on a task-specific supervision signal to make a soft selection of these topic-specific vectors. another important question is how clms should be used to obtain contextualized word vectors. given a sentence mentioning w, a model such as bert-base constructs 12 vector representations of w, i.e. one for each layer of the transformer stack. previous work has suggested to use the average of particular subsets of these vectors. in particular, @xcite found that lexical semantics is most prevalent in the representations from the early layers, and that averaging vectors from the first few layers seems to give good results on many benchmarks. on the other hand, these early layers are least affected by the sentence context @xcite , hence such strategies might not be suitable for learning topic-specific vectors. we therefore also explore a different strategy, which is to mask the target word in the given sentence, i.e. to replace the entire word by a single @xcite token, and to use the vector representation of this token at the final layer. the resulting vector representations thus specifically encode what the given sentence reveals about the target word, making this a natural strategy for learning topic-specific vectors. note that there is a clear relationship between this latter strategy and cbow @xcite : where in cbow the vector representation of w is obtained by averaging the vector representations of the context words that co-occur with w, we similarly represent words by averaging context representations. the main advantage compared to cbow thus comes from the higher-quality context encodings that can be obtained using clms. the main challenge, as already mentioned, is that we cannot consider all the mentions of w, whereas this is typically feasible for cbow (and other standard word embedding models). our contributions can be summarized as follows foot_0 : ## related work a few recent works have already proposed strategies for computing static word vectors from clms. while @xcite relied on principal components of individual transformer layers for this purpose, most approaches rely on averaging the contextualised representations of randomly selected mentions of the target word @xcite . several authors have pointed out that the representations obtained from early layers tend to perform better in lexical semantics probing tasks. however, @xcite found that the optimal layer depends on the number of sampled mentions, with later layers performing better when a large number of mentions is used. rather than fixing a single layer, @xcite advocated averaging representations from several layers. note that none of the aforementioned methods uses masking when computing contextualized vectors. this means that the final representations may have to be obtained by pooling different word-piece vectors, usually by averaging them. as an alternative to using topic models, chronis and erk (2020) cluster the contextual word vectors, obtained from mentions of the same word. the resulting multi-prototype representation is then used to compute word similarity in an adaptive way. along similar lines, @xcite cluster contextual word vectors for word sense induction. @xcite showed that clustering the contextual representations of a given set of words can produce clusters of semantically related words, which were found to be similar in spirit to lda topics. the idea of learning topic-specific representations of words has been extensively studied in the context of standard word embeddings @xcite @xcite . to the best of our knowledge, learning topic-specific word representations using clms has not yet been studied. more broadly, however, some recent methods have combined clms with topic models. for instance, @xcite use such a combination for predicting semantic similarity. in particular they use the lda or gsdmm topic distribution of two sentences to supplement their bert encoding. finally, @xcite suggested using sentence embeddings from sbert @xcite as input to a neural topic model, with the aim of learning more coherent topics. ## constructing word vectors in section 3.1, we first describe different strategies for deriving static word vectors from clms. section 3.2 subsequently describes how we choose the most relevant topics for each word, and how we sample topic-specific word mentions. finally, in section 3.3 we explain how the resulting topicspecific representations are combined to obtain task-specific word vectors. ## evaluation we compare the proposed strategy with standard word embeddings and existing clm-based strategies. in section 4.1 we first describe our experimental setup. section 4.2 then provides an overview of the datasets we used for the experiments, where we focus on lexical classification benchmarks. these benchmarks in particular allow us to assess how well various semantic properties can be predicted from the word vectors. the experimental results are discussed in section 4.3 and a qualitative analysis is presented in section 4.4. ## conclusions we have proposed a strategy for learning static word vectors, in which topic models are used to help select diverse mentions of a given target word and a contextualized language model is subsequently used to infer vector representations from the selected mentions. we found that selecting an equal number of mentions per topic substantially outperforms purely random selection strategies. we also considered the possibility of learning a weighted average of topic-specific vector representations, which in principle should allow us to "tune" word representations to different tasks, by learning task-specific topic importance weights. however, in practice we found that a standard average of the topic specific vectors leads to a comparable performance, suggesting that the outperformance of our vectors comes from the fact that they are obtained from a more diverse set of contexts.