Unsupervised Improvement of Factual Knowledge in Language Models (2023)

Abstract: Masked language modeling (MLM) plays a key role in pretraining large language models. But the MLM objective is often dominated by high-frequency words that are sub-optimal for learning factual knowledge. In this work, we propose an approach for influencing MLM pretraining in a way that can improve language model performance on a variety of knowledge-intensive tasks. We force the language model to prioritize informative words in a fully unsupervised way. Experiments demonstrate that the proposed approach can significantly improve the performance of pretrained language models on tasks such as factual recall, question answering, sentiment analysis, and natural language inference in a closed-book setting.

## introduction pretrained language models (plms) such as bert @xcite , roberta @xcite , bart @xcite , t5 @xcite use a masked language modeling (mlm) objective during pretraining. however, a traditional mlm objective may not be optimal for knowledgeintensive tasks @xcite . it has been shown that language models can benefit from incorporating knowledge within the training objective in the form of entity embeddings @xcite , knowledge retriever @xcite , knowledge embedding @xcite or augmented pretraining corpora created from knowledge graphs @xcite . despite their effectiveness, these approaches rely on existing knowledge bases and entity embeddings to incorporate knowledge within the training objective. these resources are expensive to construct and may not be available for all languages and domains @xcite . in this work, we propose a pretraining approach that can achieve better performance on knowledgeintensive tasks without using any existing knowl-edge base. we combine two key strategies to influence mlm objective. firstly, the tokens with higher informative relevance should be masked more frequently @xcite . secondly, mistakes on informative tokens should be penalized more severely. the informative relevance of the tokens can be computed efficiently with a onepass computation on the pretraining corpora. experiments demonstrate that the proposed training strategy can help the language model achieve better performance on the factual knowledge recall benchmark lama @xcite , extractive question answering (qa) benchmark squad @xcite , prompt based sentiment analysis and natural language inference (nli) tasks in autoprompt @xcite . the key contribution of this work is proposing a completely unsupervised stand-alone mlm pretraining objective for language models that can significantly improve performance on knowledgeintensive tasks. unlike prior works in the area, our method does not require existing knowledge bases to incorporate knowledge during pretraining. we make the code publicly available. foot_0 2 related work plms as knowledge bases it has been shown that large-scale plms such as bert can be used as a knowledge base @xcite . prior works have focused on factual knowledge with regards to generative plms @xcite , multilingual setting @xcite , entities and query types @xcite , fact checking @xcite . ## methodology we use mlm objective for pretraining, similar to prior works @xcite @xcite . given a sequence of tokens z, a subset of tokens x ⊂ z is randomly sampled for replacement (|x|/|z| ≈ 0. 15 in devlin et al. (2019) ). for the replacement candidates in x, 80% of the time the replacement is done with a special token [mask], 10% of tokens are replaced with a random token, and the other 10% of candidates are left unchanged @xcite @xcite . the task of the model during pretraining is to predict the original tokens from the modified input sequence. for a set of replaced tokens x(x 1 , x 2 , ..., x n ) and their corresponding output tokens y (y 1 , y 2 , ..., y n ), the loss l m lm is computed as follows: here, x i,j is the logit produced for output candidate j given input x i and v is the vocabulary set. in traditional mlm loss computation, a uniform penalty is applied for all tokens within the vocabulary. in our work, we try to influence the mlm objective during pretraining to incorporate more factual knowledge. we differ from traditional mlm pretraining in two ways: (a) instead of masking all tokens with equal probability, we allow some tokens to be masked more frequently if they have higher informative relevance, (b) we use weighted cross entropy loss to penalize mistakes on some tokens more severely if they have higher informative relevance. simple illustrations of these two concepts are shown in figure 1 . we compute the loss as follows: in this context, the informative relevance of tokens represents how important a particular token is with regard to the factual knowledge. tokens that are more important for factual knowledge (e.g. named entities) are expected to have a higher informative relevance. we use pointwise mutual information (pmi @xcite ) to compute informative relevance in an unsupervised manner. we hypothesize that words that have high pmi with their neighboring words tend to have higher informative relevance. firstly, we compute word co-occurrence statistics for the pretraining corpus within a skipgram window. secondly, pmi between all word pairs within the vocabulary is computed. thirdly, we consider the pairwise pmi between all words within a particular document in the form of a matrix (as shown in figure 2 ), so that the row-wise sum in that matrix reflects the token-specific informative relevance within that document. then informative relevance for a token is averaged across the corpus. finally, the computed values are normalized and converted to token-specific masking rates and token-specific penalty weights. those masking rates are used to create masked inputs and the penalty weights are then incorporated during mlm loss computation, as shown in equation 2. ## conclusion in this work, we propose a pretraining strategy that can be effective in storing factual knowledge within language models. the additional knowledge helps the model outperform previous approaches on a variety of knowledge-intensive nlp tasks, such as factual recall, zero-shot qa, closed-book sentiment analysis, and natural language inference. our model also achieves better performance when fine-tuned on squad and glue tasks. in the future, we aim to extend our work for text-to-text pretrained models such as t5 @xcite .