Learning Efficient Task-Specific Meta-Embeddings with Word Prisms (2020)

Abstract: Word embeddings are trained to predict word cooccurrence statistics, which leads them to possess different lexical properties (syntactic, semantic, etc.) depending on the notion of context defined at training time. These properties manifest when querying the embedding space for the most similar vectors, and when used at the input layer of deep neural networks trained to solve downstream NLP problems. Meta-embeddings combine multiple sets of differently trained word embeddings, and have been shown to successfully improve intrinsic and extrinsic performance over equivalent models which use just one set of source embeddings. We introduce word prisms: a simple and efficient meta-embedding method that learns to combine source embeddings according to the task at hand. Word prisms learn orthogonal transformations to linearly combine the input source embeddings, which allows them to be very efficient at inference time. We evaluate word prisms in comparison to other meta-embedding methods on six extrinsic evaluations and observe that word prisms offer improvements in performance on all tasks.

## introduction a popular approach to representing word meaning in nlp is to characterize a word by "the company that it keeps" @xcite . this intuition is the basis of famous word embedding techniques such as word2vec @xcite and glove @xcite . however, the question of what company a word keeps -i.e., what should define a word's context -is open. a word's context could be defined via a symmetric window of 1, 2, 5, 10, 20 words, the words that precede it, the words that follow it, the words with which it shares a dependency edge, etc. determining the utility of such different notions of context for training word embeddings is a problem that has attracted considerable attention @xcite @xcite @xcite but there is no conclusive evidence that any single notion of context could be the best for solving nlp problems in general. thus, many deep learning solutions for nlp have yet another hyperparameter to tune: what set of word embeddings should be selected for the input layer of the model. as nlp tasks become more and more complex, the practice of providing a deep model with only one notion of a word's meaning becomes limiting. word meta-embeddings address aspects of this problem by proposing techniques for combining multiple sets of word embeddings before providing them into the input layer of a downstream model. @xcite motivated word meta-embeddings by arguing that they are advantageous for the following reasons: diversity -combining embeddings trained with different algorithms on different corpora will allow for more distinct meanings of the words to persist; and, coverage -combining embeddings trained on different corpora help to better solve the out-of-vocabulary problem. however, they did not acknowledge that different sets of word embeddings can be diverse even when trained on the same corpus with the same algorithm, so long as their context windows are different. additionally, due to the various practical and theoretical similarities between different algorithms @xcite ; figure 1 : word prisms ( §3) during supersense tagging ( §4). we display the nearest neighbor to the embedding for the word "apple" in five of the window-based facets (figure 2 , §3.2) used in the downstream model ( §4). we include the ⊥ symbol on the learned transformation, p f , to indicate its orthogonality. @xcite , the gains to be found in diversifying at the level of algorithmic variation are likely to be minimal. with regard to vocabulary coverage, the out-of-vocabulary problem is at least partially addressed by character n-gram based embedding algorithms such as fasttext @xcite and subword-based decomposition techniques that can be applied post-training @xcite . nonetheless, meta-embeddings have been shown to consistently outperform models that use only a single set of embeddings in their input layer. our goal is to determine how to best combine many sets of input embeddings in order to obtain high quality results in downstream tasks. this work proposes word prisms as a simple and general way to produce and understand metaembeddings, visualized in figure 1 . word prisms excel at combining many sets of source embeddings, which we call facets. they do so by learning task-specific orthogonal transformations to map embeddings from their facets to the common meta-embedding space. this produces a vector space that is more disentangled than the original space of facet embeddings. it allows the combination of multiple source embeddings while preserving most information within each embedding set. to our knowledge, this work is the first to incorporate both explicit orthogonal transformations of source embeddings and importance weights for source embedding sets that are dynamically learned with the downstream tasks in the same meta-embedding method. furthermore, it is the first to explore combining so many sets of source embeddings (thirteen). we compare the word prisms method to other standard meta-embedding algorithms (averaging (coates and bollegala, 2018), concatenation @xcite , and dynamic meta-embeddings, dmes, @xcite ). word prisms overcome the shortcomings of each of these algorithms: (1) in averaging, performance deteriorates considerably when there are many facets -the orthogonal transformations in word prisms resolve this problem; (2) concatenation and dmes are too expensive during inference when there are many facets -word prisms only need the final meta-embeddings at inference time, making them as efficient as averaging. our results demonstrate that neural downstream models using word prisms generally obtain better results than the other algorithms across six downstream tasks, including supersense tagging, pos tagging, named entity recognition, natural language inference, and sentiment analysis. in our ablation studies, we find that our method improves performance on downstream tasks even when the vocabulary is the same across nine source facets trained on the same corpus that differ only by the definition of context window (see figure 2 ); performance further improves by incorporating four more sets of off-the-shelf facets. 2 related work pre-trained word embedding algorithms use word cooccurrence statistics from a training corpus to map words to a low-dimensional vector space such that words with similar meanings are mapped to similar points in vector space. however, changing the embedding algorithm, training corpus, or definition of cooccurrence can have a strong impact on the resulting embeddings. consequently, word metaembedding algorithms have been developed to combine multiple embedding sets. ## word prisms and meta-embeddings in this section we introduce meta-embeddings, word prisms, and the source embeddings they are composed with in this work. a meta-embedding combines pre-trained embeddings from multiple sources (e.g., from glove and fasttext), which we call facets. we define w f as the embedding of word w in facet f , for each facet @xmath0, ..., f }. the dimensionality of each embedding in a facet is denoted d f , and the final dimensionality of the meta-embedding is d . the following equation represents the general form of all meta-embedding variants and baselines considered in this work: that is, the meta-embedding for a word w, meta(w) ∈ r d is constructed as follows: first, it is projected by a linear transformation (learned or fixed) characterized by a matrix p f and bias b f for some set of embedding facets. next, the meta-embedding is a linear combination of the transformed embeddings scaled by some (learned or fixed) weights, α f . the vocabulary of a word prism is the union of the vocabularies of its facets. if a word is out-of-vocabulary for a facet f , we assign its representation w f to be the centroid of the embeddings in the facet. ## experiments our experiments seek to determine if: (1) word prisms offer improvements over the other common meta-embedding methods; and, (2) if it is desirable to produce meta-embeddings with many different notions of context from the same corpus. for (1), we pursue a variety of experiments comparing word prisms to the following meta-embedding methods: the averaging baseline, the concatenation baseline, and dynamic meta-embeddings (dmes) @xcite . for (2), we experiment with several sets of meta-embedding facet combinations. the first set is fasttext and glove (fg), as is done by @xcite . the second set is a combination of 13 different facets (all), as detailed in §3.2. in §5 we present our main results, and in §6 we present an ablation study to determine the impact of other meta-embedding combinations and the transformation matrices in word prisms. ## ablation studies we perform two ablation studies to further inspect the impact of the orthogonal transformations and meta-embedding combination choice in word prisms. we compare the orthogonal transformation in word ## conclusion in this paper, we study a simple and efficient method for constructing meta-embeddings from wideranging facets while preserving individual invariance with orthogonal transformations. the effectiveness of the proposed word prisms is validated by six supervised extrinsic evaluation tasks. our word prism models obtain consistent improvements over dynamic meta-embeddings @xcite and the averaging and concatenation baselines @xcite in all six tasks. analysis of the transformed embeddings suggests the "natural clustering" hypothesis for representation learning @xcite is important to consider for combining various source embeddings to create performant taskspecific meta-embeddings. several future directions present themselves from this work. first, we believe that contextualized embedding models can benefit from prismatic representations of their input embeddings @xcite , and that word prisms can benefit from including contextualized embeddings as facets. second, we expect that word prisms can improve performance in other tasks such as automatic summarization, which often use a single set of word embeddings in their input layers @xcite . third, we believe that meta-embeddings and the method behind word prisms can be generalized past word-based representations to sentence representations @xcite and may improve their quality, as was recently demonstrated by @xcite . lastly, recent work has found simple word embeddings to be useful for solving diverse problems from the medical domain @xcite , to materials science @xcite , to law @xcite ; we expect that word prisms and their motivations can further improve results in these applications.