Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (2020)

Abstract: Unsupervised pretraining models have been shown to facilitate a wide range of downstream NLP applications. These models, however, retain some of the limitations of traditional static word embeddings. In particular, they encode only the distributional knowledge available in raw text corpora, incorporated through language modeling objectives. In this work, we complement such distributional knowledge with external lexical knowledge, that is, we integrate the discrete knowledge on word-level semantic similarity into pretraining. To this end, we generalize the standard BERT model to a multi-task learning setting where we couple BERT’s masked language modeling and next sentence prediction objectives with an auxiliary task of binary word relation classification. Our experiments suggest that our “Lexically Informed” BERT (LIBERT), specialized for the word-level semantic similarity, yields better performance than the lexically blind “vanilla” BERT on several language understanding tasks. Concretely, LIBERT outperforms BERT in 9 out of 10 tasks of the GLUE benchmark and is on a par with BERT in the remaining one. Moreover, we show consistent gains on 3 benchmarks for lexical simplification, a task where knowledge about word-level semantic similarity is paramount, as well as large gains on lexical reasoning probes.

## introduction unsupervised pretraining models, such as gpt and gpt-2 @xcite , elmo @xcite , and bert @xcite yield state-of-the-art performance on a wide range of natural language processing tasks. all these models rely on language modeling (lm) objectives that exploit the knowledge encoded in large text corpora. bert @xcite , as one of the current state-of-the-art models, is pretrained on a joint objective consisting of two parts: (1) masked language modeling (mlm), and (2) next sentence prediction (nsp). through both of these objectives, bert still consumes only the distributional knowledge encoded by word co-occurrences. while several concurrent research threads are focused on making bert optimization more robust @xcite or on imprinting external world knowledge on its representations @xcite @xcite @xcite , no study yet has been dedicated to mitigating a severe limitation that contextualized representations and unsupervised pretraining inherited from static word embeddings: every model that relies on distributional patterns has a tendency to conflate together pure lexical semantic similarity with broader topical relatedness @xcite . in the past, a plethora of models have been proposed for injecting linguistic constraints (i.e., lexical knowledge) from external resources to static word embeddings @xcite @xcite in order to emphasize a particular lexical relation in a specialized embedding space. for instance, lexically informed word vectors specialized for pure semantic similarity result in substantial gains in a number of downstream tasks where such similarity plays an important role, e.g., in dialog state tracking @xcite or for lexical simplification @xcite . existing specialization methods are, however, not directly applicable to unsupervised pretraining models because they are either (1) tied to a particular training objective of a static word embedding model, or (2) predicated on the existence of an embedding space in which pairwise distances can be modified. in this work, we hypothesize that supplementing unsupervised lm-based pretraining with clean lexical information from structured external resources may also lead to improved performance in language understanding tasks, especially in tasks where distinguishing between pure semantic similarity and conceptual relatedness is paramount. we propose a novel method to inject linguistic constraints, available from lexico-semantic resources like wordnet @xcite and babelnet @xcite , into unsupervised pretraining models, and steer them towards capturing word-level semantic similarity. to train the lexically informed bert (libert), we (1) feed semantic similarity constraints to bert as additional training instances and (2) predict lexico-semantic relations from the constraint embeddings produced by bert's encoder @xcite . in other words, libert adds lexical relation classification (lrc) as the third pretraining task to bert's multi-task learning framework. we compare libert to a lexically blind "vanilla" bert on the glue benchmark @xcite ) and report their performance on corresponding development and test portions. libert yields performance gains over bert on 9/10 glue tasks (and is on a par with bert on the remaining one), with especially wide margins on tasks involving complex or rare linguistic structures such as diagnostic natural language inference and linguistic acceptability. moreover, we assess the robustness and effectiveness of libert on 3 different datasets for lexical simplification (ls), a task proven to benefit from word-level similarity specialization @xcite . we report ls improvements of up to 8.2% when using libert in lieu of bert. for direct comparability, we train both libert and bert from scratch, and monitor the gains from specialization across iterations. interestingly, these do not vanish over time, which seems to suggest that our specialization approach is suitable also for models trained on massive amounts of raw text data. ## language understanding evaluation to isolate the effects of injecting linguistic knowledge into bert, we train base bert and libert in the same setting: the only difference is that we additionally update the parameters of libert's transformer encoder based on the gradients of the lrc loss l lrc from eq. ( 2 ). in the first set of experiments, we probe the usefulness of injecting semantic similarity knowledge on the well-known suite of glue tasks @xcite , while we also present the results on lexical simplification, another task that has been shown to benefit from lexico-semantic similarity specialization @xcite , later in §5. ## similarity-oriented downstream evaluation: lexical simplification task description. the goal of lexical simplification is to replace a target word w in a context sentence s with simpler alternatives of equivalent meaning. generally, the task can be divided into two main parts: (1) generation of substitute candidates, and (2) candidate ranking, in which the simplest candidate is selected @xcite . unsupervised approaches to candidate generation seem to be predominant lately @xcite . in this task, discerning between pure semantic similarity and broad topical relatedness (as well as from other lexical relations such as antonymy) is crucial. consider the example: "einstein unlocked the door to the atomic age," where unlocked is the target word. in this context, the model should avoid confusion both with related words (e.g. repaired) and opposite words (e.g. closed) that fit in context but alter the original meaning. experimental setup. in order to evaluate the simplification capabilities of libert versus bert, we adopt a standard bert-based approach to lexical simplification @xcite , dubbed bert-ls. it exploits the bert mlm pretraining task objective for candidate generation. given the complex word w and a context sentence s, we mask w in a new sequence s . next, we concatenate s and s as a sentence pair and create the bert-style input by running wordpiece tokenization on the sentences, adding the [cls] and [sep] tokens before, in-between, and after the sequence, and setting segment ids accordingly. we then feed the input either to bert or libert, and obtain the probability distribution over the vocabulary output by the mlm predictor based on the masked token p(•|s, s \{w}). based on this, we select the candidates as the top k words according to their probabilities, excluding morphological variations of the masked word. for the substitution ranking component, we also follow @xcite . given the set of candidate tokens c, we compute for each c i in c a set of features: (1) bert prediction probability, (2) loss of the likelihood of the whole sequence according to the mlm when choosing c i instead of w, (3) semantic similarity between the fasttext vectors @xcite of the original word w and the candidate c i , and (4) word frequency of c i in the top 12 million texts of wikipedia and in the children's book test corpus. 8 based on the individual features, we next rank the candidates in c and consequently, obtain a set of ranks for each c i . the best candidate is chosen according to its average rank across all features. in our experiments, we fix the number of candidates k to 6. ## conclusion and future work we have presented libert, a lexically informed extension of the state-of-the-art unsupervised pretraining model bert. our model is based on a multi-task framework that allows us to steer (i.e., specialize) the purely distributional bert model to accentuate a lexico-semantic relation of true semantic similarity (as opposed to broader semantic relatedness). the framework combines standard bert objectives with a third objective formulated as a relation classification task. the gains stemming from such explicit injection of lexical knowledge into pretraining were observed for 9 out of 10 language understanding tasks from the glue benchmark, as well as for 3 lexical simplification benchmarks. these results suggest that complementing distributional information with lexical knowledge is beneficial for unsupervised pretraining models. in the future, we will work on more sophisticated specialization methods, and also investigate the use of adapter modules @xcite @xcite for more efficient fine-tuning. we will also explore methods to encode the knowledge on asymmetric relations such as meronymy and lexical entailment. finally, we will port this new framework to other languages and other pretrained language models @xcite @xcite . we also plan to experiment with applications of our approach to resource-poor scenarios in the vein of @xcite , and also to specialized domains (e.g., (bio)medicine, law) @xcite , where the transferred and/or external in-domain lexical information might prove particularly useful.