Distilling Word Meaning in Context from Pre-trained Language Models (2021)

Abstract: In this study, we propose a self-supervised learning method that distils representations of word meaning in context from a pre-trained masked language model. Word representations are the basis for context-aware lexical semantics and unsupervised semantic textual similarity (STS) estimation. A previous study transforms contextualised representations employing static word embeddings to weaken excessive effects of contextual information. In contrast, the proposed method derives representations of word meaning in context while preserving useful context information intact. Specifically, our method learns to combine outputs of different hidden layers using self-attention through self-supervised learning with an automatically generated training corpus. To evaluate the performance of the proposed approach, we performed comparative experiments using a range of benchmark tasks. The results confirm that our representations exhibited a competitive performance compared to that of the state-of-the-art method transforming contextualised representations for the context-aware lexical semantic tasks and outperformed it for STS estimation.

## introduction word representations are the basis for various natural language processing tasks. particularly, they are crucial as a component in context-aware lexical semantics and in the estimation of unsupervised semantic textual similarity (sts) @xcite @xcite . word representations are desired to represent word meaning in context to improve these downstream tasks. large-scale masked language models pre-trained on massive corpora, e.g., bi-directional encoder representations from transformers (bert) @xcite , embed both the context and meaning of a word; thus, word-level representations generated by such masked language models are called contextualised word representations. previous studies @xcite have revealed that lexical information and contextspecific information are captured in different layers of masked language models. they argued that a sophisticated mechanism is required to derive representations of word meaning in context from them. although contextualised word representations have shown considerable promise, how best to compose the outputs of different layers of masked language models to effectively represent word meaning in context remains an open question. @xcite improved contextualised word representations by transforming their space towards static word embeddings, e.g., fasttext @xcite . although this transformation is computationally efficient, the process is monotonic, weakening the effect of context in representations. as an orthogonal approach, pre-trained masked language models should fit themselves to generate representations of word meaning in context with supervised fine-tuning. however, annotating word meanings in context is non-trivial, and no such resource is abundantly available. to address these challenges, we propose a method that distils representations of word meaning in context from masked language models via self-supervised learning. 1 specifically, our model combines the outputs of different hidden layers using a self-attention mechanism @xcite . the distillation model is self-supervised using an autoencoder to reconstruct original representations with an automatically generated training corpus. in contrast to the transformation-based approach, our representations preserve useful context information intact. experimental results on a range of benchmark tasks show that our representations exhibited a performance competitive with that of the state-of-the-art method that transforms contextualised representations for context-aware lexical semantics. furthermore, the results confirm that our representations are more effective for composing sentence representations, which contributes to unsupervised sts estimation. ## distilling word meaning in context inspired by the representation disentanglement approach (section 2.2), we model the distillation of representations of word meaning in context using an autoencoder framework, as shown in figure 1 . @xcite probed pretrained language models for lexical semantic tasks, revealing that lexical information is scattered across lower layers, whereas context-specific information is embedded in higher layers. hence, we aim to distil the outputs of different hidden layers using a transformer layer. in this study, although we adopted bert as the masked language model, the proposed method is directly applicable to other pre-trained models. figure 1 shows the model architecture. first, we obtain the outputs of all hidden layers of a masked language model, mlm(•), with frozen parameters @xmath0, where s is an input sentence of length |s| containing the target word, w @xmath1, ℓ is the number of hidden layers in the masked language model (0 corresponding to its embedding layer), and d is the hidden dimension of the masked language model. we then extract the outputs of the hidden layers corresponding to the target word, w t , from h, noting that when w t is segmented into a set of m sub-words ω 1 , ω 2 , • • • , ω m , by a tokeniser of the masked language model, we compute the layer-wise averages of the hidden outputs of all sub-words @xcite . that is, h @xmath2 we then input these hidden outputs into a meaning distillation model to derive a representation for word meaning in context. we also input the hidden outputs to another distillation model that derives information other than word meaning in context. for convenience, hereinafter we refer to this information as the context and the distillation model as the context distillation model. 2 each distillation model consists of a transformer layer followed by a mean-pooling function to obtain meaning and context representations, expressed as h @xmath3, respectively. finally, we reconstruct the original representation from h m and h c . although there are different approaches for reconstructions, such as using a neural-network-based decoder, a sophisticated decoder may learn to fit itself to mimic the masked language model outputs. hence, we adopt meanpooling as the simplest reconstruction mechanism for reconstruction. the reconstruction target @xmath4. for inference, we use h m as a representation of word meaning in context. averaging the outputs of the layers in the tophalf of masked language models consistently performs well for context-aware lexical semantic tasks @xcite . thus, we set @xmath5. 3 john @xcite reported that a variational autoencoder (kingma and welling, 2014) outperformed the simpler autoencoder on representation disentanglement. however, this was not the case in this study, wherein the autoencoder consistently outperformed the variational version. we intend to further investigate auto-encoding architectures in future work. ## self-supervised learning the meaning and context distillation models described in section 3 require constraints to ensure that the desired attributes are distilled; otherwise, these distillation models obtain a degenerate solution that simply copies the original representations. we design a self-supervision framework ensuring that word meaning in context is distilled using an automatically generated training corpus. ## experimental setup we empirically evaluated whether our method distils representations of word meaning in context from a masked language model using contextaware lexical semantic tasks and sts estimation tasks. 4 all the experiments were conducted on an nvidia tesla v100 gpu. we compared our method to @xcite as the state-of-the-art in the family of methods that transform contextualised representations. recall that @xcite adopt an approach orthogonal to that proposed herein, which transforms word representations from the masked language model using static word embeddings. specifically, we used fasttext as the static embeddings that performed most robustly across models and tasks. as a baseline, we also show the performance of bert. based on the previous studies @xcite , we used the average of the outputs of the top-half layers, i.e., equation (1), which consistently performed well in lexical semantic tasks. ## results and discussions below, we discuss experimental results and the results of in-depth analyses conducted to identify characteristics of meaning representations generated by our method. ## summary and future work we have proposed a method that improves contextualised word representations. the proposed approach distils a representation of word meaning in context, retaining useful context information encoded by a masked language model. experimental results confirmed that our method exhibited performance competitive with the state-of-the-art method for transforming contextualised representations to alleviate excessive effects of contexts on representations, demonstrated on context-aware lexical semantic tasks. our method further outperformed it on sts tasks. in a future work, we plan to investigate correspondences of the context representations. we had assumed that these representations preserve the sentence-level meaning; however, the sts results confirmed that this assumption was incorrect. another possibility is that context representations may retain syntactic information. we intend to conduct in-depth investigations using syntactic tasks. moreover, we will expand our method to support multilingual masked language models to contribute to cross-lingual processing, e.g., cross-lingual word in context disambiguation @xcite , word alignment @xcite , and quality estimation and post-editing for machine translation @xcite .