Self-supervised Post-processing Method to Enrich Pretrained Word Vectors (2023)

Abstract: Retrofitting techniques, which inject external resources into word representations, have compensated for the weakness of distributed representations in semantic and relational knowledge between words. However, the previous methods require additional external resources and strongly depend on the lexicon. To address the issues, we propose a simple extension of extrofitting, self-supervised extrofitting: extrofitting by its own word vector distribution. Our methods improve the vanilla embeddings on all of word similarity tasks without any external resources. Moreover, the method is also effective in various languages, which implies that our method will be useful in lexicon-scarce languages. As downstream tasks, we show its benefits in dialogue state tracking and text classification tasks, reporting better and generalized results compared to other word vector specialization methods.

## introduction static word vectors are still widely used in natural language tasks despite the recent trends of contextualized models. for example, in a wide study of dialogue state tracking (dst) @xcite , contextualized models performed worse than static word embeddings. it seems reasonable that the contextualized models do not perform well in such a lack of context, which implies that static embeddings are still useful. to make better word vectors, we focus on retrofitting ideas (also called word vector postprocessing), which injects the semantic information from external resources by modifying the values of pretrained word vectors @xcite ; @xcite ; inter alia). the benefits of post-processing methods are that (1) the methods can reflect additional resources into the word vectors without re-training, (2) the methods can be applied to any kinds of pretrained word vectors, and (3) retrofitting can make word vectors specialize in a specific task. the previous studies focusing on explicit retrofitting have used manually defined or learned functions to make synonyms close and antonyms distant (see the details in §4). as a result, their approaches strongly rely on external resources. furthermore, we agree that making synonyms close together is somewhat reasonable, even though the synonyms have different nuances in some contexts. however, other kinds of relations, such as antonyms, should be further investigated. for instance, love and hate are generally grouped as antonyms. most of the previous studies have made the words distant from each other, but the words definitely share the meaning of emotion in their representations. we thus conjecture that the methods are not generalized well. in this paper, we propose word vector enrichment based on extrofitting @xcite ): our methods consistently improve model performances in the fundamental tasks. in contrast, other post-processing methods degrade the performance. ## preliminary extrofitting extrofitting @xcite expands word embedding matrix w by concatenating w with r w : next, trans calculates transform matrix given the matrix w: in the end, extrofitting is formulated as follows: ## self-supervised extrofitting we consider the word embedding matrix as the term-document matrix; the x-axis of the matrix is vocabulary and the y-axis is (unknown) semantic dimension. thus, as researchers have use the decomposition of the term-document matrix to get the term-vectors in lsa, the decomposition of the term-semantic matrix can be considered to represent term-vectors (us ) and semantic-vectors (s v t ). this intuition corresponds to the traditional methods in vector embedding, verified in @xcite ; @xcite . with the idea, we first decompose word embeddings matrix w to make latent representations as follows: as described above, we can get term-vectors, which are word representations in k-dimensional latent space, by computing u k s k . adjusting the dimension of latent space (k in §2), we calculate semantically related words using cosine similarity. for every word, we calculate its cosine similarity to all other words in the vocabulary. if the similarity exceeds a predetermined threshold, we group the words. note that a group can contain only a single word. this process is repeated iteratively for each ungrouped word until all words are clustered. we use the set of semantically related words (c') as the class (synonym pairs, c) of extrofitting instead of semantic lexicons: to sum up, we use the principle of vanilla extrofitting @xcite , which utilizes lda algorithms to group in-domain instances (in this case, word vectors). simultaneously, the algorithm pushes out-domain instances apart. instead of using pre-defined lexicons for extrofitting, we use the idea of lsa to get semantically related words. even if the number of extracted word pairs are small or the words are not meaningful, the process of making out-domain instances far can make better representations. in the experiments, we start with pretrained glove @xcite ) (if we do not mention it explicitly) and set a threshold that determines whether a pair of words are semantically related. we use a high threshold (0.9 of cosine similarity) since type ii error is rather better than type i error. fied word vectors by weighted averaging the word vectors with semantic lexicons. extending from the simple idea, counter-fitting @xcite used both synonym pairs to collect word vectors and antonym pairs to make word vectors the gene from one another. next, paragram embeddings @xcite used synonyms and negative sampling to collect the word vectors. borrowing attract-terms from the paragram embeddings and adding repel-terms, attract-repel @xcite injected linguistic constraints into word vectors through predefined cost function with mono-/cross-lingual linguistic constraints. explicit retrofitting (er-cnt) @xcite directly learned mapping functions of linguistic constraints with deep neural network architectures. they then used the functions to retrofit the word vectors. post-specialization @xcite resolved the problem of the previous models that only updates words in external lexicons. some works have used cross-lingual resources to get further semantic information @xcite . while the previous methods utilized text-level resources, vecmap @xcite used other word embeddings as external resources. 2 for fair comparisons, we input the same word vectors for source and target embeddings of the method. recently, bert-based model lexfit @xcite was derived, but the method requires external resources. we use 6 datasets in english: wordsim353 (w(s) for similarity and w(r) for relation) @xcite , rareword (rw) @xcite , men-3k (me) @xcite , semeval (se) @xcite , simlex-999 (sl) @xcite , and simverb-3500 (sv) @xcite . ## discussion the specialized word vectors do not warrant better performance. we conjecture that the methods trade off catastrophic forgetting against semantic specialization even though the previous methods successfully specialize their vectors into given lexicons, losing the pretrained information encoded in the original glove. it affects the performance of the fundamental tasks, which are largely degraded. on the other hand, our method enriches the word vector in general, resulting in marginal but certain improvements. contextual representations provide substantial improvements in nlp tasks, but the representations lack semantic information due to their nature. since the context and the semantics are different types of information that can enrich representations, we believe our approaches might further improve the contextual representations. ## conclusion we develop a self-supervised retrofitting model that enriches word vectors without semantic lexicons. the method utilizes its own distribution of word vectors to get better representations. in the exp. 1, we show that our method can improve the performance on word similarity tasks and present qualitative examples. it can be applied to other pretrained word vectors, resulting in better performances on all the word similarity datasets. selfextro also has potential advantages in lexicon-scarce languages. in the exp. 2 and 3, we presented the effect of post-processing methods on downstream tasks. our method shows marginal but certain improvements, while other post-processed word vectors largely degrade the performances, which seems the result of losing generalization.