S ense BERT : Driving Some Sense into BERT (2020)

Abstract: The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the ‘Word in Context’ task.

## introduction neural language models have recently undergone a qualitative leap forward, pushing the state of the art on various nlp tasks. together with advances in network architecture @xcite , the use of self-supervision has proven to be central to these achievements, as it allows the network to learn from massive amounts of unannotated text. the self-supervision strategy employed in bert @xcite involves masking some of the words in an input sentence, and then training the model to predict them given their context. other proposed approaches for self-supervised objectives, including unidirectional @xcite , permutational @xcite , or word insertionbased @xcite methods, operate similarly, over words. however, since a given word form can possess multiple meanings (e.g., the word 'bass' can refer to a fish, a guitar, a type of singer, etc.), the word itself is merely a surrogate of its actual meaning in a given context, referred to as its sense. indeed, the word-form level is viewed as a surface level which often introduces challenging ambiguity @xcite . in this paper, we bring forth a novel methodology for applying weak-supervision directly on the level of a word's meaning. by infusing wordsense information into bert's pre-training signal, we explicitely expose the model to lexical semantics when learning from a large unannotated corpus. we call the resultant sense-informed model sensebert. specifically, we add a maskedword sense prediction task as an auxiliary task in bert's pre-training. thereby, jointly with the standard word-form level language model, we train a semantic-level language model that predicts the missing word's meaning. our method does not require sense-annotated data; self-supervised learning from unannotated text is facilitated by using wordnet @xcite , an expert constructed inventory of word senses, as weak supervision. we focus on a coarse-grained variant of a word's sense, referred to as its wordnet supersense, in order to mitigate an identified brittleness of finegrained word-sense systems, caused by arbitrary sense granularity, blurriness, and general subjectiveness @xcite . word-net lexicographers organize all word senses into 45 supersense categories, 26 of which are for nouns, 15 for verbs, 3 for adjectives and 1 for adverbs (see full supersense table in the supplementary materials). disambiguating a word's supersense has been widely studied as a fundamental lexical categorization task @xcite @xcite . we employ the masked word's allowed supersenses list from wordnet as a set of possible labels for the sense prediction task. the labeling of words with a single supersense (e.g., 'sword' has only the supersense noun.artifact) is straightforward: we train the network to predict this supersense given the masked word's context. as for words with multiple supersenses (e.g., 'bass' can be: noun.food, noun.animal, noun.artifact, noun.person, etc.), we train the model to predict any of these senses, leading to a simple yet effective soft-labeling scheme. we show that sensebert base outscores both bert base and bert large by a large margin on a supersense variant of the semeval word sense disambiguation (wsd) data set standardized in @xcite . notably, sensebert receives competitive results on this task without funetuning, i.e., when training a linear classifier over the pretrained embeddings, which serves as a testament for its self-acquisition of lexical semantics. furthermore, we show that sensebert base surpasses bert large in the word in context (wic) task (pilehvar and camacho-collados, 2019) from the superglue benchmark @xcite , which directly depends on word-supersense awareness. a single sensebert large model achieves state of the art performance on wic with a score of 72.14, improving the score of bert large by 2.5 points. ## related work neural network based word embeddings first appeared as a static mapping (non-contextualized), where every word is represented by a constant pretrained embedding @xcite . such embeddings were shown to contain some amount of word-sense information @xcite @xcite . additionally, sense embeddings computed for each word sense in the word-sense inventory (e.g. wordnet) have been employed, relying on hypernymity relations @xcite or the gloss for each sense @xcite . these approaches rely on static word embeddings and require a large amount of annotated data per word sense. the introduction of contextualized word embeddings @xcite , for which a given word's embedding is context-dependent rather than precomputed, has brought forth a promising prospect for sense-aware word embeddings. indeed, visualizations in @xcite show that sense sensitive clusters form in bert's word embedding space. nevertheless, we identify a clear gap in this abilty. we show that a vanilla bert model trained with the current word-level self-supervision, burdened with the implicit task of disambiguating word meanings, often fails to grasp lexical semantics, exhibiting high supersense misclassification rates. our suggested weakly-supervised word-sense signal allows sensebert to significantly bridge this gap. moreover, sensebert exhibits an improvement in lexical semantics ability (reflected by the word in context task score) even when compared to models with wordnet infused linguistic knowledge. specifically we compare to @xcite who re-contextualize word embeddings via a wordto-entity attention mechanism (where entities are wordnet lemmas and synsets), and to @xcite which construct sense embeddings from bert's word embeddings and use the word-net graph to enhance coverage (see quantitative comparison in ## incorporating word-supersense information in pre-training in this section, we present our proposed method for integrating word sense-information within sense-bert's pre-training. we start by describing the vanilla bert architecture in subsection 3.1. we conceptually divide it into an internal transformer encoder and an external mapping w which translates the observed vocabulary space into and out of the transformer encoder space [see illustration in figure 1(a) ]. in the subsequent subsections, we frame our contribution to the vanilla bert architecture as an addition of a parallel external mapping to the words supersenses space, denoted s [see illustration in figure 1(b) ]. specifically, in section 3.2 we describe the loss function used for learning s in parallel to w , effectively implementing word-form and wordsense multi-task learning in the pre-training stage. then, in section 3.3 we describe our methodology for adding supersense information in s to the initial transformer embedding, in parallel to word-level information added by w . in section 3.4 we address the issue of supersense prediction for out-ofvocabulary words, and in section 3.5 we describe our modification of bert's masking strategy, prioritizing single-supersensed words which carry a clearer semantic signal. ## semantic language model visualization devlin et al. (2019)), has an immediate non-trivial bi-product. the pre-trained mapping to the supersenses space, denoted s, acts as an additional head predicting a word's supersense given context [see figure 1(b)]. we thereby effectively attain a semantic-level lan-sensebert base semeval-ss fine-tuned 30k no oov 81.9 30k average oov 82.7 60k no oov 83 sensebert's semantic language model allows predicting a distribution over supersenses rather than over words in a masked position. figure 3(a) shows the supersense probabilities assigned by sensebert in several contexts, demonstrating the model's ability to assign semantically meaningful categories to the masked position. bert sensebert noun.artifact noun.group noun.quantity noun.body same different ten shirt-sleeved ringers stand in a circle, one foot ahead of the other in a prize-fighter's stance sent. a: the kick must be synchronized with the arm movements. a sidecar is a smooth drink but it has a powerful kick. ## lexical semantics experiments in this section, we present quantitative evaluations of sensebert, pre-trained as described in section 3. we test the model's performance on a supersense-based variant of the semeval wsd test sets standardized in @xcite , and on the word in context (wic) task (pilehvar and camacho-collados, 2019) (included in the recently introduced superglue benchmark @xcite ), both directly relying on the network's ability to perform lexical semantic categorization. ## conclusion we introduce lexical semantic information into a neural language model's pre-training objective. this results in a boosted word-level semantic awareness of the resultant model, named sensebert, which considerably outperforms a vanilla bert on a semeval based supersense disambiguation task and achieves state of the art results on the word in context task. this improvement was obtained without human annotation, but rather by harnessing an external linguistic knowledge source. our work indicates that semantic signals extending beyond the lexical level can be similarly introduced at the pre-training stage, allowing the network to elicit further insight without human supervision.