DMLM : Descriptive Masked Language Modeling (2023)

Abstract: Over the last few years, Masked Language Modeling (MLM) pre-training has resulted in remarkable advancements in many Natural Language Understanding (NLU) tasks, which sparked an interest in researching alternatives and extensions to the MLM objective. In this paper, we tackle the absence of explicit semantic grounding in MLM and propose Descriptive Masked Language Modeling (DMLM), a knowledge-enhanced reading comprehension objective, where the model is required to predict the most likely word in a context, being provided with the word’s definition. For instance, given the sentence “I was going to the _”, if we provided as definition “financial institution”, the model would have to predict the word “bank”; if, instead, we provided “sandy seashore”, the model should predict “beach”. Our evaluation highlights the effectiveness of DMLM in comparison with standard MLM, showing improvements on a number of well-established NLU benchmarks, as well as other semantics-focused tasks, e.g., Semantic Role Labeling. Furthermore, we demonstrate how it is possible to take full advantage of DMLM to embed explicit semantics in downstream tasks, explore several properties of DMLM-based contextual representations and suggest a number of future directions to investigate.

## introduction language modeling is at the core of transfer learning approaches that have recently revolutionized the natural language processing field. among these, the masked language modeling (mlm) formulation introduced by @xcite has been used to train large language models that obtained astounding performances in many natural language understanding (nlu) tasks. this proved empirically that training a model to predict a word based on the context in which it appears (i.e., the cloze task; @xcite enables the emergence of rich word representations with transferable value @xcite . given the importance of mlm, several improvements over its standard formulation have been proposed. in particular, a large body of research has investigated alternative self-supervised objectives to take advantage of the ever-growing availability of raw text. for instance, @xcite unified multiple language modeling objectives into a single architecture, @xcite masked entire spans instead of single tokens, and @xcite exploited the entire input instead of only optimizing on the masked words. at the same time, another research direction has explored utilizing, alongside mlm, the wealth of information contained in structured knowledge bases (kbs) to enhance models' representations. for example, @xcite and @xcite tried leveraging kb entities in order to provide additional input context, while @xcite and @xcite tasked the models with explicitly predicting kb-grounded embeddings of concepts and named entities in place of words, as an integration to the mlm framework. our work stands in the middle of the aforementioned directions. indeed, we have designed a semantic-enhanced objective that is able to semantically ground word representations and provide complementary information to the cloze task without ever leaving the mlm framework. specifically, we put forward descriptive masked language modeling (dmlm), a pre-training objective that requires the model to perform reading comprehension over textual definitions: given an input sentence, the model is tasked with predicting a masked word in context while being provided with a natural language definition, drawn from a predefined sense inventory, that describes its meaning. at the same time, given that this auxiliary task i went @xcite the @xcite . sandy @xcite beach to seashore i went @xcite the @xcite . financial @xcite bank to institution figure 1 : two examples of dmlm, where a model has to predict a described masked word while simultaneously performing standard mlm. to summarize, our contributions are manifold: we release code as well as all sense-tagged corpora used for training at https://github.com/ sapienzanlp/dmlm . ## related work since the advent of bert @xcite , masked language modeling (mlm) has been used widely to pre-train language models in a selfsupervised fashion. as opposed to standard language modeling, which requires predicting the next word in a sequence given the preceding words, mlm consists in predicting masked words given the remaining context. ## descriptive mlm as a first step, let us formally define the task of masked language modeling (mlm): given a sequence of n words w 1 , . . . , w n , we randomly replace a certain percentage of the words by means of a special, uninformative, [mask] token, and ask the model to predict the corresponding masked words. for example, given the sentence "i went to the beach.", if we picked the word beach randomly, the model would see "i went to the @xcite ." as input, and would be asked to predict beach from its corresponding [mask] token. dmlm builds on top of mlm in that, before applying mlm, i) it randomly selects a content word w i from the input sentence, for which we know its textual definition d w i , ii) it replaces w i with another special token, i.e., @xcite , and, iii) it appends d w i to our input sequence after a special [define] token. going back to our example, we would obtain "i went to the [def]. [define] sandy seashore". after this initial step, we apply the standard mlm perturbation avoiding replacing either of the two special tokens that we added, but leaving d w i as a possible target for perturbation, e.g., "i went [mask] the [def]. [define] sandy [mask]" (see figure 1 ). following this procedure, depending on which tokens are masked, the model has to leverage, potentially simultaneously, both the input sequence and the content word's definition in order to restore the masked words. thus, utilizing dmlm implies that: dmlm can be used to pre-train both encoderonly and encoder-decoder architectures. for encoder-only architectures, when a content word foot_1 is split into subwords, we replicate the [def] token for each subword so that the model is able to reconstruct the full word at prediction time. for encoderdecoder architectures, dmlm's formulation can easily be applied with very small adjustments. due to space and computational constraints, we discuss these adjustments in more detail in appendix a. ## dmlm pre-training training a system using the dmlm objective requires a large corpus whose content words are inventory tokens instances token vocab. definitions wordnet 98 m 44 m 224 k 89 k ode 98 m 72 m 84 k 79 k wiktionary 98 m 69 m 183 k 91 k following the literature, we use wordnet 3.0; following the reference paper, we trained es-cher jointly on all three inventories, obtaining results that were comparable to the original model. 4 using the resulting model, we tagged the entire wikitext-103 corpus, at sentence level, three times, once for each inventory, which we posit acts as a regularization factor for the dmlm objective. indeed, the model might encounter the same content word in the exact same sentence but with different definitions representing the same meaning, foot_4 thus reducing overfitting on the definitions themselves. in the end, the disambiguated corpus contained around 3.8m sentences with a total of approximately 381k unique definitions coming from the three different inventories. ## experimental evaluation in order to assess the quality of the trained models, we evaluated them on a number of different, both semantic and non-semantic, downstream tasks. each of the following subsections, aside from section 5.1 which describes the comparison systems, contains the setup and results for each experiment we performed. ## conclusions in this work, we presented an extension of mlm called descriptive masked language modeling (dmlm), which embeds semantic information via natural language descriptions in the pre-training phase of language models. we found that, under the tested settings, dmlm consistently outperforms mlm on multiple benchmarks. on the glue benchmark, a set of natural language understanding tasks, we observed improvements on both semantic and non-semantic tasks. furthermore, using srl as a proxy, we also demonstrated two important properties of models trained with dmlm: first, that it is possible to leverage dmlm's pre-training objective to consistently improve performances in downstream tasks and, second, that the model can generalize to definitions that were not seen during pre-training. finally, we discovered that, even without any explicit signal towards spatial alignment, the output space of a dmlm-trained encoder tends to relate to the knowledge bases used to retrieve the definitions. we posit that this might be a very desirable property for better handling ambiguity, e.g., in machine translation, where recent works have shed some light on the issue of semantic biases @xcite . additionally, in principle, we could make dmlm-trained systems more suitable for domain-specific tasks. for example, in the medical domain, we could impose precise meanings for word senses based on a healthcare-specific knowledge base and thus reduce the conflation of senses' representation in the same space. second, being able to drive spatial representations of words during the training and select a reference knowledge base as a guide might be the very reason why dmlmtrained systems outperform, at least in our experimental setting, mlm-trained systems. given the encouraging results obtained while scaling up the network size, and to foster research in this direction, we release our code and the sensetagged wikitext-103 corpus.