Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling (2023)

Abstract: In this paper, we propose an alternative to the classic masked language modeling (MLM) pre-training paradigm, where the objective is altered from the reconstruction of the exact identity of randomly selected masked subwords to the prediction of their latent semantic properties. We coin the proposed pre-training technique masked latent semantic modeling (MLSM for short). In order to make the contextualized determination of the latent semantic properties of the masked subwords possible, we rely on an unsupervised technique which uses sparse coding. Our experimental results reveal that the fine-tuned performance of those models that we pre-trained via MLSM is consistently and significantly better compared to the use of vanilla MLM pretraining and other strong baselines.

## introduction recent successes in natural language processing are predominantly fueled by the use of large pretrained language models (plms) that are constructed in a self-supervised manner over massive amounts of raw text. autoencoder-style language models @xcite @xcite ; inter alia) are typically trained via masked language modeling (mlm). plms pre-trained using mlm are capable of returning distributions over their vocabulary that peak at plausible substitutes of masked (sub)tokens given some sequence of input text. the individual updates performed during mlm pre-training, however, are not aligned to what we expect from the plms in the long run, i.e., that they output distributions of plausible substitutes for masked (sub)tokens. as a motivating example, consider the sentence "alice is eating a cake.", and suppose that we randomly select the token cake to be masked. for such a training example, we would obtain the masked input sentence "alice is eating a @xcite .". the smallest possible training loss would incur for this particular example if our model allocated all its output probability mass to the word "cake" as being the only possible replacement of the [mask] token, while assigning precisely zero probability to other alternatives, that are otherwise totally viable from a human cognitive perspective, including words such as pear, croissant, soup, etc. what eventually provides plms trained with the mlm objective the ability to output token distributions that are plausible from a human perspective, is that they are trained over massive amounts of diverse batches, and the different possible substitutes even out in expectation. the hypothesis that we investigate in this paper is that we can train plms more efficiently if -instead of relying on the exact identity of the masked tokens during pre-trainingwe required our model to output such distributions that are not peaked at a single symbol (corresponding to the identity of the masked token). a more natural and perhaps more sample efficient option to overcome the misalignment between the individual pre-training updates of plms and their long-term objective could compare the output distribution of the model to some desired distribution of substitutes, which would -instead of encoding the masked input token with a one-hot categorical distribution -assign nonzero probabilities to multiple viable tokens. note, however, if we had access to such desired output token distributions for the masked tokens, then language modeling was already solved and the task of training plms would become obsolete. an alternative approach for pre-training that mitigate the exact reliance on the identity of the masked tokens could rely on semantic resources, such as wordnet @xcite and conceptnet @xcite . in this case, one might require the language model to output semantic properties of the masked tokens, i.e., in the previous example input sentence, instead of recovering the word cake for the [mask] token, the goal of the plm was to output its semantic properties, such that the masked word refers to a concept that is edible and is a kind of dessert. one difficulty of such an approach is that obtaining semantic resources with sufficient coverage is notoriously arduous @xcite . in addition, such a pre-training paradigm would need the training corpus to be annotated with the possible properties of the words in their particular contexts. few well-resourced languages have such sense-annotated corpora available, e.g., semcor @xcite for english, but their size is still several orders of magnitude smaller compared to the size of the corpora used for pre-training plms. in this work, we propose masked latent semantic masking (mlsm), such an alternative of mlm, which attenuates the direct reliance on the masked tokens during pre-training. the main idea behind mlsm is that we no longer require the plm to recover the exact identity of the masked tokens, but we predict their (latent) semantic properties. our pre-training procedure is based on the observation that sparse representations obtained via sparse coding on the hidden representations of neural models are well aligned with human interpretable properties @xcite @xcite . the way we incorporate the above property into mlsm pre-training is that we derive context sensitive latent semantic information about the masked tokens by performing sparse coding on their hidden representation and we aim at predicting those as a pre-training task. since we determine the sparse representations in an unsupervised way, our approach is not affected by the difficulties that would arose when relying on external semantic resources. our evaluation confirms our expectations, i.e., that plms pre-trained with mlsm can significantly outperform such models that are trained via vanilla mlm and other strong baselines. we release our code for performing mlsm pre-training at https://github.com/szegedai/mlsm . ## related work integrating external semantic knowledge into plms has gained increasing research interest @xcite @xcite @xcite @xcite @xcite . these efforts cover a wide method-ological spectrum, depending on how the external knowledge is incorporated into the plms. the different approaches can be distinguished for instance on the location where the external knowledge gets injected, i.e., either at the input, architectural or output level. colon-hernandez et al. (2021) provides a comprehensive overview of approaches aiming at the incorporation of external knowledge into plms. what all these approaches have in common is that they rely on some explicit knowledge representation, e.g., in the form of triplets of some knowledge graph. in contrast, our main research question was whether it is possible to increase the semantic awareness of plms without having access to explicitly stored knowledge during the pre-training. sensebert @xcite ) is such a modification of bert that also aims at the integration of semantic knowledge, however, it also differs from our approach in multiple aspects. the most important difference is that mlsm does not require any external linguistic resources, whereas sense-bert relies on wordnet, making it only available for languages where such a linguistic resource is accessible. our fully unsupervised approach for inducing implicit semantic information to words within their context builds on the observation that performing sparse coding over the contextual representations of plms results in such sparse contextualized representations that align well with the semantic categories of the words. @xcite showed that these sparse vectors can be used for improving word sense disambiguation (wsd), whereas @xcite used it for creating visualizations that help understanding the inner workings of plms from a semantic perspective. @xcite provided further evidence that sparse contextualized word representations can be successfully exploited in cross-lingual wsd as well. ## methodology as opposed to vanilla mlm, being agnostic to the semantics of the masked tokens, we propose mlsm, an alternative pre-training formulation which gets enhanced via semantic information. ## experiments we pre-trained several bert language models of medium size relying on the transformers library @xcite . the medium models @xcite comprise of 8 transformer blocks and use a hidden representation of 512 dimensions. we considered the official pretrained bert-base-cased model as the teacher model t for mlsm and we relied on its tokenizer for all the models that we pre-trained. in order to assess the effects of the different pre-training variants, we evaluated the fine-tuning performance of the models over 10 diverse tasks. we repeated all fine-tuning experiments 10 times in order to account for the variability of the results that occur due to the random initialization of the taskspecific classification head @xcite . we used 3 nvidia a6000 gpus for performing the experiments. each pre-training and the repeated fine-tuning experiments that followed them for a single checkpoint took approximately 2 gpu weeks and 1 gpu day, respectively. foot_0 ## conclusion in this paper, we proposed masked latent semantic modeling (mlsm), such a modification of the classical masked language modeling task, where the goal is changed from the prediction of the exact identity of the masked tokens to that of their latent semantic categories. we suggested a contextsensitive unsupervised approach for determining the latent semantic categories of tokens by performing sparse coding of their hidden representations from a pre-trained teacher model. the reliance on a teacher model makes our approach similar to model distillation in the sense that we can transfer the capabilities of a (larger) pre-trained model into a newly trained one. comparison of the fine-tuning capabilities of the plm pre-trained via classical model distillation and mlsm revealed a clear benefit towards the latter approach. our experiments also corroborate that mlsm pre-training behaves more sample efficient compared to other alternatives, as the fine-tuning performance of the pre-trained model at its 25% completeness level was capable of achieving better performances than any of the alternatively trained models at their end of pre-training stage (see figure 2 ). more importantly, our experiments revealed that by relying on mlsm pre-training, it was possible to cram the fine-tuning capabilities of a plm with 2.5× parameter into a smaller one (see