Task-dependent Optimal Weight Combinations for Static Embeddings (2022)

Abstract: A variety of NLP applications use word2vec skip-gram, GloVe, and fastText word embeddings. These models learn two sets of embedding vectors, but most practitioners use only one of them, or alternately an unweighted sum of both. This is the first study to systematically explore a range of linear combinations between the first and second embedding sets. We evaluate these combinations on a set of six NLP benchmarks including IR, POS-tagging, and sentence similarity. We show that the default embedding combinations are often suboptimal and demonstrate 1.0-8.0% improvements. Notably, GloVes default unweighted sum is its least effective combination across tasks. We provide a theoretical basis for weighting one set of embeddings more than the other according to the algorithm and task. We apply our findings to improve accuracy in applications of cross-lingual alignment and navigational knowledge by up to 15.2%.

## introduction static word embeddings are used in a broad range of nlp applications, including conversational gameplay @xcite ), text categorization @xcite , translation @xcite @xcite , affordance detection @xcite , and semantic analysis @xcite . in addition to using static embeddings directly, researchers often combine them with contextualized models or use them for embedding initialization of downstream tasks @xcite such as summarization @xcite and neural machine translation @xcite . the persistence of static embeddings is due in part to their ease of use and low computational requirements. rather than needing a forward pass through a neural network to embed each word, pre-trained embeddings can be stored in memory and retrieved with complexity 𝑂 (1). foot_0 §4.2 outlines more advantages of static embeddings. we propose an augmentation of three popular embedding methods (word2vec skip-gram, glove, and fast-text). word2vec skip-gram @xcite is a neural word context predictor, glove @xcite is a log-bilinear model that includes global context information with a co-occurrence matrix, and fast-text @xcite incorporates sub-word information via character n-grams with a skip-gram objective to expedite training and handle unseen words. more details about these algorithms are in §2. each of them produces two separate embedding sets ("target" and "context", see figure 1 ) that we combine in previously unexplored ways. we show that these typically unexplored target and context combinations reveal much about embedding effectiveness. our key contributions are as follows: we demonstrate up to 12.5% improvements over baseline performance on a diverse set of nlp benchmarks by combining target and context vectors. @xcite we analyze embeddings statistically and show that (1) word2vec target vectors encode better word-to-word relationships while context vectors are better suited for bag-of-words representations, (2) glove default vectors perform well on tasks for which they were tuned but under-perform generally, and (3) fasttext target vectors' sub-word encodings are useful in many tasks but counterproductive for bag-of-words representations. (see §6.) finally, in §7 we employ our methods practically to improve performance on muse @xcite cross-lingual alignment by 0.69-1.56% and navigational robotics benchmarks by up to 15.2% ## background we overview three common static embedding algorithms: word2vec skip-gram with negative sampling, fasttext, and glove. each of these algorithms has the same output: a set of embedding vectors, each corresponding to a word in a vocabulary. we outline applications that employ these different algorithms, but note that any task that uses vectors from one of these algorithms could just as feasibly use the vectors from another of them. @xcite word2vec skip-gram model learns embeddings as a neural regression problem: predicting each word's context. each word in question is called the target, and its neighbors are called context. the model learns two sets of embeddings, corresponding to target and context words, though only the first is typically used. word2vec is employed in many tasks, including measurement of mwe candidates @xcite and epidemic-related twitter stream classifications @xcite . in our experiments we used skip-gram with negative sampling (rather than mikolov et al.'s cbow model) because of its comparability to glove and fasttext. (see §3.) throughout the text we refer to this algorithm simply as "word2vec. " the fasttext algorithm @xcite ) integrates sub-word information into the skip-gram framework. it embeds character n-grams, and a word's embedding is the sum of its sub-word vectors. the context vectors are not composed of sub-words. many applications use fasttext, including hyperbolic word representations @xcite and low-resource sentence similarity @xcite . popular glove embeddings @xcite are used for sarcasm detection @xcite , emotion detection @xcite , and lexical semantic analysis @xcite ). glove's log-bilinear model learns two embeddings from word co-occurrence. by default, most public glove implementations sum the embeddings evenly. however, our study shows that this unweighted sum often does not maximize performance. (see §5.) our work challenges the assumption that the default combinations or selections of target and context vectors (target only for word2vec and fasttext and an unweighted sum for glove) are optimal for any task, or even across tasks in general. we methodically explore a spectrum of target/context combinations for each algorithm and show that the default embedding selection is often not the best. although not directly studied here, other static embedding algorithms such as conceptnet numberbatch @xcite , hyperbolic word embeddings @xcite , and word2vec-cbow @xcite exist and merit study. ## theoretical motivation this section shows that while the target and context vectors produced by the glove algorithm are fundamentally equivalent to each other, the same does not hold true in the cases of word2vec and fasttext. this insight informs and motivates our analysis of various context/target embeddings in §6. ## methodology we conduct experiments to answer three core questions: (1) how does task performance vary across linear combinations of target/context vectors, and do the default settings work generally well? (2) is the pattern of performance (as a function of target/context weightings) similar in all three embedding algorithms? if not, what are the differences? (3) are optimal weighting schemes data-and task-specific? if so, to what extent? to answer these, we trained target and context embeddings for each of three algorithms (word2vec skipgram, glove, fasttext) on six corpora. we then produced more embeddings by combining each pair of target t and context c as follows: foot_1 .8𝑡 + .2𝑐, .6𝑡 + .4𝑐, 𝑡 + 𝑐, .4𝑡 + .6𝑐, and .2𝑡 + .8𝑐. we refer to these combinations respectively as 80:20-sum, 60:40-sum, true sum, 40:60sum, and 20:80-sum, in addition to target and context vectors alone. we generated 126 embedding sets total (six training corpora × three embedding algorithms × seven weighted sums). this spread of linear combinations has not been studied previously. we insist that the target and context weights sum to 1 for the sake of uniformity and clarity in drawing conclusions from our experiments. since our nlp applications employ magnitude-agnostic cosine distance as a similarity metric, allowing the weights to range from 0 to 1 is largely equivalent to letting them range from 0 to 𝑁 for any positive real number 𝑁 . we set 𝑁 = 1 across all experiments to allow fair comparisons of weight values. it is worth noting that weights could also be negative, a possibility that is beyond the scope of our current study but that could be explored in future work. linear combinations are a form of summation. summation is an established method for combining vector information, as discussed in §2.2. the fasttext algorithm relies on sums of subword target vectors @xcite . glove vectors are often targetcontext sums by default @xcite . target and context vectors ought to be reasonably well aligned for summation, since the objective of a static embedding model is to increase the dot products between target and context vectors, and the dot product is based on component-wise multiplication. this provides a theoretical motivation for our focus on target-andcontext vector summation. note, however, that summation is not the only possible way to combine target and context embeddings. as discussed in §2.2, fulda and robinson (2021) explored concatenating the embeddings instead. however, they found that summation yielded better results, a conclusion that they verified both theoretically and experimentally. despite the theoretical basis for their alignment, target and context vectors may not be perfectly aligned for summation in practice. one could employ a more intricate approach to align them, such as a linear mapping or an encoder/decoder reconstruction method such as muse @xcite . this is a potentially promising area of future research. note that our experimentation over a range of seven different target/context weighting values is comparable to performing a course grid search for this parameter in each task. another more computationally expensive and task-specific method to tune this parameter is meta-learning. our objective in our main experiments is to learn generalizable principles across a variety of tasks, which may be of value to other researchers, for which purpose we determined that this grid search approach would suffice. however, we do explore an application of optimizing the weighting via differentiation. see §7.2 for analysis. training hyperparameters we used embedding dimension 300. for glove: we used window size 10, minimum word count 5, and 25 training iterations. these have been shown to achieve optimal results @xcite . the "minimum word count" mentioned is minimum frequency for a word in the corpus to include it in the model's vocabulary. for fasttext: we used window size 10, minimum word count 5, 5 training epochs, and 3-6 character n-grams, as standard @xcite . for word2vec: we used window size foot_2 5 and 3 epochs, as recommended by @xcite . due to the web scraped corpus' size and computing restraints, we opted for minimum word count 100. ## analysis this section provides theoretical backing for the observed performance of target vs. context vectors on specific types of tasks. we analyze the advantages of using weighted target and context combinations for specific use cases and offer recommendations for best practices in static embedding research. ## applications based on our findings, we recommend that practitioners tune the target/context embedding weight for downstream applications. the following two examples demonstrate that by doing so, it is possible to improve upon default vector performance using applications from published literature. note that our baselines differ from published results, which employed downloadable pre-trained embeddings that were extensively optimized. this approach yields high-quality results, but it does not allow for comparison of target/context combinations since only one combination is typically released online. we therefore used our own embeddings, trained on the wikipedia corpus, to explore relative performance across a spectrum of target/context combinations. we encourage future practitioners to release both embedding sets. ## conclusion by leveraging unconventional combinations of target and context vectors learned by glove, fasttest, and word2vec, we achieve improvements of up to 12.5% on common word embedding tasks such as pos-tagging and ir, thus elevating the usefulness of these popular and inexpensive word representations for nlp tasks. experiments with 126 embedding sets on six generic tasks and two downstream applications show that tuning the hyperparameter of target and context weight for downstream tasks can improve performance significantly over default settings, increasing accuracy by 0.69% to 1.56% on muse cross-lingual alignment and by up to 15.2% on navigational robotics benchmarks. analysis suggests that target-heavy word2vec combinations are most suited to tasks involving single-word relationships, while context vector information is useful in summed sentence representations. we further observe that glove default settings perform best on tasks for which glove was tuned but tend to perform poorly on others, and that fasttext target vectors excel in tasks such as pos-tagging, where sub-word information is particularly relevant. these findings reveal a disconnect between the maximum potential of static embedding algorithms and the ways in which they are typically used. in a majority of cases, the performance of pre-trained word embeddings could be improved by tuning the target/context weight hyperparameter. furthermore, because a target/context weighting is typically chosen prior to the release of extensively pre-trained word vectors, the possibility of exploring various target/context weightings has typically not been made available to subsequent researchers. in alignment with our results, we urge those who design and train static word embedding models to release both target and context vector sets. the software and embeddings used in our experiments will be released publicly under the mit license. given the widespread use of glove, word2vec, fasttext, and other static embeddings, there is a need for deeper understanding of target and context interactions. directions of future work in this area include the semantic content contained in context and target embeddings; the interplay between embedding algorithm, corpus size, and corpus genre; vector normalization methods to avoid norm imbalance; distance metrics not based on cosine similarity; and paired-embedding algorithms where context and target spaces are used individually.