Integrating Quasi-symbolic Conceptual Knowledge into Language Model Pre-training (2024)

Abstract: In this paper, we investigate the integration of latent conceptual knowledge into the pre-training of masked language models. Our solution is based on the use of an auxiliary model, from which we extract training signals for training a student model. We determine the training signals from the hidden representations of the student model in an unsupervised way, using sparse coding. Models trained on latent concepts alone have an improved fine-tunability on downstream tasks, however, they perform worse on traditional language modeling, i.e., when the goal is to output missing tokens as opposed to latent semantic classes of words. In order to preserve the improved fine-tuning capability of the models, while making them better at the task of language modeling, we propose a final stage of pre-training, during which we perform traditional masked language modeling. The final stage of pre-training is based on a model that has already been pre-trained on the task of modeling latent semantic properties, with the weights of the backbone model being frozen. During the final training phase, we only train a lightweight linear classifier layer on top of the logits that the model determines for the latent semantic properties. With this modification, we can obtain the benefits of both the traditional training paradigms and the one which is based on the use of latent semantic properties. We release our source code at github.com/SzegedAI/MLSM .

## introduction language acquisition involves forming a rich battery of concepts and the ability to use and manipulate those concepts. even though human cognition is rooted in concepts, this is not reflected in the typical pre-training of neural language models. in contrast, standard pre-training techniques ignore the concept-oriented nature of language when they expect a single ground truth token to be predicted during pre-training time. @xcite argues for the need of integrating conceptual information into language models, while @xcite recommended a knowledge distillation approach for doing so. the masked latent semantic modeling (mlsm) approach @xcite relies on an auxiliary teacher model that steers the pre-training of the student model by performing sparse coding on its hidden representation and requiring the student model to recover those instead of the actual tokens. as the location of the non-zero coefficients in the sparse contextualized word representations obtained that way can be viewed as quasi-symbolic latent semantic concepts @xcite , the pre-training becomes driven by concepts as opposed to tokens. while the favorable properties of mlsm pretrained models have been demonstrated in obtaining models with improved fine-tuning capabilities, models pre-trained with it struggle on tasks that require language modeling ability, i.e., predicting actual token substitutes for missing/masked token positions from an input sequence. this is a consequence of the modeling in mlsm being shifted from the actual tokens to the latent concepts determined in an unsupervised way. in this work, we extend such a modification to mlsm modeling, which ensures that the final model does not only have improved fine-tuning capability, but it is also capable of performing regular language modeling on the token level. we achieve this goal by integrating a lightweight post pretraining phase, during which we keep the weights of the model determined via mlsm fixed, and add a small a final linear module to the network (while freezing the rest of it), such that the token predictions are made on the logits that the originally pre-trained model would return towards the latent concepts. this modification ensures that the positive properties of mlsm and traditional masked language modeling (mlm) pre-training can be integrated into a single final model. ## masked latent semantic modeling we first overview the mlsm pre-training technique, as it plays a central role in our modified model architecture. the way mlsm works is that it changes the domain of the output distribution of the model from its vocabulary of subword units (as in mlm) to the inventory of quasi-symbolic latent semantic properties that we determine in an unsupervised manner. in figure 1 , we provide a visual comparison between the mlm and mlsm pre-training techniques. the way mlsm determines the latent semantic properties of some token is by relying on an already pre-trained auxiliary model t . in a preparatory phase, a representative sample of hidden representations produced by t is collected from its layer l as {h solving (1) is performed in advance to the actual pre-training, with a negligible (≪ 1%) computational overhead compared to the costs of pretraining. once the dictionary matrix d (l) is determined, it is used for determining the sparse contextualized representation for any h (l) i , i.e., a hidden state from layer l of t as arg min objective (2) is computationally convenient, as it does not require optimizing towards d (l) . with d (l) being fixed from (1), obtaining the sparse linear coefficients of α i constitutes an efficiently solvable lasso optimization problem. due to the non-negativity constraint imposed towards α i in (2), the ℓ 1 -normalized sparse linear coefficients can be conveniently treated as probability distributions over the k latent semantic concepts. transformer encoder sarah ate a delicious [mask] e(sarah) e(ate) e(a) e(delicious) e([mask]) input embeddings sarah ate a delicious cake output probability a 0.0 ⋮ ⋮ cake 1.0 ⋮ 0.0 ⋮ ⋮ zyzzyva 0.0 (a) masked language modeling (mlm) 0.00 ⋮ 0.61 0.00 ⋮ 0.39 0.00 -1.21 ⋮ 0.51 ⋮ 0.08 @xmath0. ## pre-training our proposed pre-training consists of three sequential steps. in the first step, we used classical masked language modeling for pre-training. the models pre-trained at this stage serve as the auxiliary teacher model for the subsequently trained model (see figure 1b ). as mlsm does not output subtokens, it is expected to have limited capabilities in performing tasks that require outputting distributions over the vocabulary of the model. we trained a separate unigram tokenizer for the two corpora, with a 25,000 vocabulary size. during all three stages of pre-training, we used the adamw optimizer with a peak learning rate of 0.0001 and an effective batch size of 1024 (that resulted from using gradient accumulation over 8 batches). when masking is involved, we employ the typically chosen 15% random masking rate selected dynamically from the batches. ## experimental results we evaluate our models using the official evaluation framework of the shared task that was provided by the organizers @xcite . the evaluation involved model fine-tuning on various glue tasks @xcite as well as zero-shot evaluations towards the blimp @xcite and ewok @xcite benchmarks. ## conclusions in this paper, we investigated the integration of latent concepts extracted from an auxiliary model into the sample efficient pre-training of neural language models. we gave multiple modifications to existing approaches, including a masking-free variant of the originally proposed approach and the inclusion of a final, lightweight pre-training phase into the pre-training procedure, which ensures that the final model is not only capable of modeling semantic properties of tokens, but it can also accurately predict the identity of masked word form based on the latent semantic properties that the backbone model determines. finally, we make the models that we pre-trained openly accessible from https://huggingface.co/szegedai (the models named with prefix babylm24). sciences. the research received additional support from the european union project rrf-2.3.1-21-2022-00004 within the framework of the artificial intelligence national laboratory.