Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors (2020)

Abstract: One of the most powerful features of contextualized models is their dynamic embeddings for words in context, leading to state-of-the-art representations for context-aware lexical semantics. In this paper, we present a post-processing technique that enhances these representations by learning a transformation through static anchors. Our method requires only another pre-trained model and no labeled data is needed. We show consistent improvement in a range of benchmark tasks that test contextual variations of meaning both across different usages of a word and across different words as they are used in context. We demonstrate that while the original contextual representations can be improved by another embedding space from both contextualized and static models, the static embeddings, which have lower computational requirements, provide the most gains.

## introduction word representations are fundamental in natural language processing (nlp) @xcite . recently, there has been a surge of contextualized models that achieve state-of-the-art in many nlp benchmark tasks @xcite @xcite . even better performance has been reported from finetuning or training multiple contextualized models for a specific task such as question answering @xcite . however, little has been explored on directly leveraging the many off-the-shelf pre-trained models to improve taskindependent representations for lexical semantics. furthermore, classic static embeddings are often overlooked in this trend towards contextualized models. as opposed to contextualized embeddings that generate dynamic representations for words in context, static embeddings such as word2vec @xcite assign one fixed representation for each word. despite being less effective in capturing context-sensitive word meanings, static embeddings still achieve better performance than contextualized embeddings in traditional contextindependent lexical semantic tasks including word similarity and analogy @xcite . this suggests that static embeddings have the potential to offer complementary semantic information to enhance contextualized models for lexical semantics. we bridge the aforementioned gaps and propose a general framework that improves contextualized representations by leveraging other pre-trained contextualized/static models. we achieve this by using static anchors (the average contextual representations for each word) to transform the original contextualized model, guided by the embedding space from another model. we assess the overall quality of a model's lexical semantic representation by two inter word tasks that measure relations between different words in context. we also evaluate on three within word tasks that test the contextual effect from different usages of the same word/word pair. our method obtains consistent improvement across all these context-aware lexical semantic tasks. we demonstrate the particular strength of leveraging static embeddings, and offer insights on the reasons behind the improvement. our method also has minimum computational complexity and requires no labeled data. ## background this section briefly introduces the contextualized/static models that we experimented in this study. for static models, we select three representative methods. sgns @xcite , as the most successful variant of word2vec, trains a log linear model to predict context words given a target word with negative sampling in the objective. fasttext improves over sgns by training at the n-gram level and can generalize to unseen words @xcite . in addition to these two prediction-based models, we also include one count-based model, glove @xcite . glove is trained to encode semantic relations exhibited in the ratios of word-word occurrence probabilities into word vectors. as opposed to static embeddings, contextualized models provide dynamic lexical representations as hidden layers in deep neural networks typically pre-trained with language modeling objectives. in our study, we choose three state-of-the-art contextualized models. bert @xcite trains bidirectional transformers @xcite with masked language modeling and next sentence prediction objectives. @xcite 's roberta further improves upon bert by carefully optimizing a series of design decisions. xl-net @xcite takes a generalized autoregressive pre-training approach and integrates ideas from transformer-xl @xcite . for the best performance, we use the large cased 1 variant for each contextualized model. since our study focuses on generic lexical representations and many of the lexical semantic tasks do not provide training data, we extract features 2 from these contextualized models without fine-tuning the weights for a specific task. this feature-based approach is also more efficient compared with finetuning the increasingly larger models which can have hundreds of millions of parameters. ## method our method 3 is built from a recently proposed cross-lingual alignment technique called meeting in the middle @xcite . their method relies on manual translations to learn a transformation over an orthogonal alignment for better cross-lingual static embeddings. we show that by a similar alignment + transformation technique, we can improve monolingual contextualized embeddings without resorting to any labeled data. the direct correspondence among contextualized and static embeddings for alignment is not straightforward, as contextualized models can compute infinite representations for infinite contexts. inspired by previous study @xcite that found contextualized embeddings roughly form word clusters, we take the average of each word's contextual representations as anchors of a contextualized model. we call them static anchors 1 1024 dimensions with case-preserving vocabulary. 2 appendix a contains more details on feature extraction. 3 implementation details are listed in appendix b. as they provide one fixed representation per word, and therefore correspond to word embeddings from a static model such as fasttext. we also use these anchors to align between contextualized models. to form the vocabulary for creating static anchors in our experiments, we take the top 200k most frequent words and extract their contexts from english wikipedia. to describe the method in more detail, we represent the anchor embeddings from the original contextualized model as our source matrix s, and the corresponding representations from another contextualized/static model as target matrix t. s i and t i are the source and target vectors for the i th word in the vocabulary (v ). we first find an orthogonal alignment matrix w that rotates the target space to the source space by solving the least squares linear regression problem in eq. 1. w is found through procrustes analysis @xcite . as described in eq. 2, we then learn a linear mapping m to transform the source space towards the average of source and the rotated target space, by minimizing the squared euclidean distance between each transformed source vector ms i and the mean vector µ i (µ @xmath0. m is the mapping we will use to transform the original contextualized space. following @xcite , m is found via a closed-form solution. for improved alignment quality, as advised by @xcite , we normalize and meancenter foot_0 the embeddings in s and t a priori. ## experiments task descriptions foot_1 we evaluate on three within word tasks. usage similarity (usim) @xcite dataset measures graded similarity of the same word in pairs of different contexts on the scale from 1 to 5. word in context (wic) (pilehvar and camacho-collados, 2019) dataset challenges a system to predict a binary choice of whether a pair of contexts for the same word belongs to the same meaning or not. we follow the advised training scheme in the original paper to learn a cosine similarity threshold on the representations. the recently proposed @xcite task provides contexts for selected word pairs from the word similarity benchmark simlex-999 @xcite and measures the graded contextual effect. we use the english dataset from this task. its first subtask, cosimlex-i, evaluates the change in similarity between the same word pair under different contexts. as it requires a system to capture different contextual representations of the same word in order to correctly predict the change of similarity to the other word in the pair, cosimlex-i indirectly measures within-word contextual effect and therefore provides our third within word task. the second cosimlex subtask, cosimlex-ii, is an inter word task as it requires a system to predict the absolute gold rating of each word pair consisting of different words in each context. we also evaluate on another related inter word task, stanford contextual word similarity (scws), which provides graded similarity scores of word pairs in independent contexts. compared with the two inter word tasks, the three within word tasks are more sensitive to contextual effects since they penalize strongly a static model (eg. fasttext) as being no better than a random baseline. by contrast, we might expect a context-independent static model to perform reasonably, though not as good as a context-sensitive model, in interw ord tasks @xcite . results: we find that applying transformation is generally able to improve each contextualized model, obtaining the best performance across all the tasks. in particular, we observe substantial improvements in usim (ca. 0.04 increase of ρ) and scws (ca. 0.03 increase of ρ). the most consistent improvement comes from leveraging static embeddings. this is especially evident in inter word tasks where transforming towards fasttext achieves the best performance but leveraging another contextualized model often brings harm. this suggests that the static embeddings are able to inject better inter-word relations @xcite into a contextualized model. at the same time, static embeddings consistently improve performance in within word tasks in 24 out of the total 27 configurations, reassuring us that the contextualization power of the original contextual space is not only preserved but even enhanced. overall, fasttext is the most robust target space as it improves all the contextualized source representations for all the tasks except for xlnet in wic. sgns and glove are also competitive especially in improving within word tasks. analysis: the overall improvement in both within word and inter word tasks suggests two possible benefits from the transformation: better withinword contextualization and better overall interword semantic space. we perform controlled studies that test for these two sources of improvement in isolation. we test on the best base contextualized space (roberta) with the various transformations. the fact that a static embedding (fasttext) performs better than a random baseline in within word tasks (see to test solely the effect on the overall inter word semantic space of the contextualized model becomes better after the transformation, we 'decontextualize' the model by evaluating only on the static anchors of the contextualized embeddings. these static anchors are not sensitive to a particular context and can thus only reflect overall inter word semantic space like embeddings from a static model. we observe improvement from the transformation on the static anchors in inter word tasks. in particular, aligning towards fasttext brings the largest and the most consistent gains. this suggests that fasttext may have offered a better ensemble space with roberta and results in a better overall inter word semantic space. to better understand the contextualization improvement in the within word scenario, we focus on wic to perform more detailed analysis. while we report results on the test set in overall, we observe a trend for the within-word contextual representations to move slightly closer to each other after the transformation, as the mean cosine similarity of a pair's contextual word representations across all instances has increased from 0.516 to 0.542. we further break down cases according to their labels and find that the transformation mainly brings the representations closer for true pairs (where the context pairs of the word are indeed expressing the same meaning) with the mean cosine similarity increased from 0.606 to 0.651. for false cases where the context pairs refer to distinct meanings of the target word, there is less increase in similarity (from 0.426 to 0.433). we also find that the improved performance in this task can be largely attributed to correcting many erroneous false predictions in the original space as these representations are drawn closer after the transformation (see appendix d). we qualitatively examine these corrected true cases (examples are provided in appendix e), and found that the improvement typically comes from reduced variance for the contextual representations of monosemous words. an example is the word daughter. we observe very low cosine similarity among its contextual representations in the original space. these representations are drawn closer after the transformation (eg. cosine similarity from 0.48 to 0.67). we suspect this might be related to contextualized models' over-sensitivity to context changes @xcite . to summarize the analysis, our controlled experiments confirm our two hypotheses that the transformation brings two independent effects: improved overall inter-word semantic space and improved within-word contextualization. our qualitative analysis shows that the improved within-word contextualization is likely to be the result of context variance reduction. ## related work it has been shown that combining different static word representations (for example through averaging or concatenation) into a meta embedding can usually lead to better lexical representations @xcite . while these task-independent meta embedding techniques are mainly applied on static embeddings, research has started to explore leveraging ensemble contextualized models when performing fine-tuning on a specific task @xcite . our method, as a post-processing transformation over task-independent contextual representations, is inherently different from these meta embedding and ensemble approaches. computationally, our method does not require maintaining multiple models at test time, and is therefore more efficient. our method is also by far the most effective way to leverage static embeddings to improve contextualized representations. 6 our methodology is related to studies on aligning cross-lingual embeddings @xcite @xcite . while these works mainly focus on obtaining better crosslingual representations, our study is the first attempt to show that some of the cross-lingual alignment methods can be applied to improve monolingual contextualized representations with no manual resources required. ## conclusion we present an effective post-processing method that transforms and enhances contextual word representations through static anchors with guidance from other contextualized/static embeddings. we show leveraging static embeddings, with no labeled data, consistently improves (across almost all configurations) on both inter word and surprisingly within word context-aware lexical semantic tasks. we also perform controlled analysis to highlight, in isolation, the improvement from the transformation on both contextualization and on an overall interword semantic space. in the future, we plan to apply the transformed representations on more lexical semantics tasks such as word sense disambiguation within an application @xcite .