Improving Contextual Representation with Gloss Regularized Pre-training (2022)

Abstract: Though achieving impressive results on many NLP tasks, the BERT-like masked language models (MLM) encounter the discrepancy between pre-training and inference. In light of this gap, we investigate the contextual representation of pre-training and inference from the perspective of word probability distribution. We discover that BERT risks neglecting the contextual word similarity in pre-training. To tackle this issue, we propose an auxiliary gloss regularizer module to BERT pre-training (GR-BERT), to enhance word semantic similarity. By predicting masked words and aligning contextual embeddings to corresponding glosses simultaneously, the word similarity can be explicitly modeled. We design two architectures for GR-BERT and evaluate our model in downstream tasks. Experimental results show that the gloss regularizer benefits BERT in word-level and sentence-level semantic representation. The GR-BERT achieves new state-of-the-art in lexical substitution task and greatly promotes BERT sentence representation in both unsupervised and supervised STS tasks.

## introduction pre-trained language models like bert @xcite and its variants @xcite @xcite have achieved remarkable success in a wide range of natural language processing (nlp) benchmarks. by pre-training on large scale unlabeled corpora, bert-like models learn contextual representations with both syntactic and semantic properties. researches show the contextual representations generated by bert capture various linguistic knowledge, including part-of-speech (pos), named entities, semantic roles @xcite @xcite , word senses @xcite , etc. furthermore, with the fine-tuning procedure, the contextual representations show excel-* the first two authors contributed equally to this work. lent transferability in downstream language understanding tasks, and lead to state-of-the-art (sota) performance. the masked language model (mlm) plays a significant role in the pre-training stage of many bert-like models @xcite . in an mlm, a token w is sampled from a text sequence s, and replaced with a [mask] token. let c be the rest of tokens in s except for w. we name c as the masked context or surrounding context, and s as the full context. during pre-training, bert encodes the masked context c into a contextual embedding vector h c , and use it to generate a contextual token probability distribution p(x|c), where @xmath0. the training objective is to predict the masked token w by maximizing likelihood function log p(w|c). in the fine-tuning or inference stage, bert takes the full context s without masks as input, and encodes every token into its contextual representation for downstream tasks. we denote the contextual representation corresponds to token w as h s . we analyze the corresponding contextual token probability distribution p(x|c) and p(x|s) generated from h c and h s , as a proxy to study the representations @xcite . figure 1(a) shows an example when masked context @xmath1, the predicted tokens with high prob-abilities p(x|c) includes good, nice, great, tough, which are all reasonable answers to the cloze task. ideally, we want the context encoder to behave the same way when full context s is given, as in figure 1 (b), the model should only propose contextual synonyms of bad such as dangerous, nasty and mean with p(x|s). however, the actual bert generates p(x|s) as shown in figure 1(c ), which contains inappropriate token proposals such as good, rough and big. the discrepancy between figure 1(b ) and 1(c) is because only the masked token distribution p(x|c) is explicitly modeled in bert with the mlm, while the full contextual token distribution p(x|s) works in an agnostic way through model generalization. this leads to a gap between p(x|c) in pretraining and p(x|s) in fine-tuning and inference. it is shown in unsupervised semantic textual similarity (sts) tasks, bert generates contextual embeddings that even underperforms static embeddings for sentence representation @xcite . although in bert pre-training, random token replacement strategy is used to mitigate the mismatch that [mask] token is never seen during fine-tuning, to the best of the authors' knowledge, there is no analysis on the gap of representation between masked context h c and full context h s in different phases when using bert. to address this issue, we perform an investigation on the inner structure of p(x|s). through theoretical derivation, we discover p(x|s) can be decomposed into the combination of masked contextual token distribution p(x|c) and a point-wise mutual information (pmi) term that describes contextual token similarity. further analysis shows both the mlm and token replacement in bert pre-training have potential shortcomings in modeling the contextual token similarity. inspired by the decomposition of p(x|s), we propose to add an auxiliary gloss regularizer (gr) module to the mlm task, where mask prediction and gloss matching are trained simultaneously in the bert pre-training. we also design two model architectures to integrate the gloss regularizer into the original mlm task. we examine our proposed model in downstream tasks including unsupervised lexical substitution (ls) @xcite , unsupervised sts and supervised sts benchmark @xcite . by invoking gloss regularized pre-training, our model improves lexical substitution task from 14.5 to 15.2 points in the ls14 dataset, leading to new sota performance. in unsupervised sts tasks, gloss regularizer improves the performance from 56.57 to 67.47 in terms of average spearman correlation by a large margin. such performance gain is also observed in supervised sts task. empirical experiments prove our model effectively generates better contextual token distribution and representations, which contributes to word-level and sentence-level language understanding tasks. ## conclusion in this work, we propose the gr-bert, a model with gloss regularization to enhance the word contextual information. we first analyze the gap between mlm pre-training and inference, and aim to model the pmi term that characterizes the word semantic similarity given context. due to the lack of data that labels the word semantic similarities given contexts, we propose to indirectly learn the semantic information in pre-training by aligning contextual word embedding space to a human annotated gloss space. we design two model structures and validate them in three nlp semantic tasks. in the lexical substitution task, we increase the sota value from 14.5 to 15.2 in ls14 best metric and many other metrics in ls07 and ls14 are also improved. in the unsupervised sts task, our gr model show its capacity in sentence representation without any training in sentence task, and it improves the mlm performance from 56.57 to 67.47. in the supervised sts-b task, gr model exceed the mlm baseline by about 0.9 points, and the gains increases to 3.44 in the low resource scenarios. our work provides a new perspective to the mlm pre-training, and show the effectiveness of modeling word semantic similarity. however, one limitation of our work is the lack of large-scale word-gloss matching data. the training data in our work is far less than that in bert pre-training, which hinders the large-scale pre-training in gr-bert. our future works will focus on mining more word-gloss training data and validate gr model in more nlp tasks. we believe there is still a big room for gr model performance improvement and possible gains in more nlp tasks. terra blevins and luke zettlemoyer. 2020. moving down the long tail of word sense disambiguation with gloss informed bi-encoders. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 1006-1017, online. association for computational linguistics. daniel cer, mona diab, eneko agirre, iñigo lopez-gazpio, and lucia specia. 2017. semeval-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation. in proceedings of the 11th international workshop on semantic evaluation (semeval-2017), pages 1-14, vancouver, canada. association for computational linguistics. ting-yun chang and yun-nung chen. 2019. what does this word mean? explaining contextualized embeddings with natural language definition. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6064-6070, hong kong, china. association for computational linguistics. a derivation of eqn. (4) and (5) by plugging eqn. (2) and (3) into eqn. (4), we have pmi(x;w|c) ≈ log p(x|w, c) p(x|c) the second term in eqn. ( 14 ) can be denoted as for the cbow model, by applying h @xmath2, the pmi function given by eqn. ( 14 ) can be transformed into