LM -Pub-Quiz: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models (2025)

Abstract: Knowledge probing evaluates to which extent a language model (LM) has acquired relational knowledge during its pre-training phase. It provides a cost-effective means of comparing LMs of different sizes and training setups and is useful for monitoring knowledge gained or lost during continual learning (CL). In prior work, we presented an improved knowledge probe called BEAR (Wiland et al., 2024), which enables the comparison of LMs trained with different pre-training objectives (causal and masked LMs) and addresses issues of skewed distributions in previous probes to deliver a more unbiased reading of LM knowledge. With this paper, we present LM-Pub-Quiz, a Python framework and leaderboard built around the BEAR probing mechanism that enables researchers and practitioners to apply it in their work. It provides options for standalone evaluation and direct integration into the widely-used training pipeline of the Hugging Face transformers library. Further, it provides a fine-grained analysis of different knowledge types to assist users in better understanding the knowledge in each evaluated LM. We publicly release LM-Pub-Quiz as an open-source project.https://lm-pub-quiz.github.io/

## introduction pre-trained language models (lms) currently take on a central role in state-of-the-art nlp approaches @xcite . given their importance, prior work has sought to measure the amount of factual knowledge encoded in lms using knowledge probing mechanisms @xcite . here, the knowledge represented in the parameters of an lm is automatically compared to factual knowledge in a relational knowledge base (kb). for instance, a probe might measure if an lm can correctly recall the capitals of countries, as illustrated in figure 1 . ## knowledge probing approach we designed lm-pub-quiz to easily assess the factual knowledge implicitly represented in a model's parameters (which is often referred to as 'parametric knowledge' or 'parametric memory'; @xcite . this knowledge may be used by the model without the need to retrieve external information and is critical to a model's ability to perform intended functions such as answering questions or writing text on specified topics @xcite . even with the increasing development and use of retrieval-augmented generation approaches, estimating the models' inherent knowledge remains critical to predict the performance of the overall system as (1) basic knowledge within the retrieval model may be required to select relevant documents in the first place, (2) good parametric knowledge can reduce how much external knowledge needs to be retrieved, (3) improve the extraction of information from retrieved content foot_1 , and (4) incorrect information in the lm may be in conflict with the retrieved information, often leading the model to disregard the retrieved information @xcite @xcite . as proposed in previous work @xcite , lm-pub-quiz ranks alternative statements (such as "the capital of uganda is kampala" and "the capital of uganda is thimphu") using the lm's (pseudo) log-likelihood. 3 if the top-ranked alternative is correct, the fact is considered to have been answered correctly (see figure 1 ). relying only on the (pseudo) log-likelihood enables the comparison of models with different pretraining objectives with little constraints on the inserted answer (i.e., they can consist of multiple tokens and be placed anywhere in the sentence). ## framework overview we give an overview of lm-pub-quiz, describe how it can be installed (section 3.1), explain the basic components of the interface (3.2), and offer examples to illustrate its usage (3.3, 3.4, & 3.5). ## example experiments to showcase example applications, we present three novel experiments that show how lm-pub-quiz can be used to conduct a detailed analysis of knowledge in lms (section 4.1 and 4.2) and how the hugging face integration (see section 4.3) can be used to monitor knowledge in a continual pre-training setting. ## comparison with existing libraries the lm evaluation harness framework @xcite is one of the most well-known evaluation tools for large language models, featuring numerous benchmarks as part of its task suite, including knowledge tasks such as mmlu @xcite . this framework primarily focuses on autoregressive language models and lacks support for mlms. this limits the ability to compare the performance of different types of models on the same datasets. similarly, helm @xcite , llm-facteval @xcite , and the llmeval module in openfactcheck @xcite rely on the capability of clms to generate continuations to a prompt and are therefore not applicable to mlms. the unique feature of lm-pub-quiz is its focused approach to cloze statement filling, allowing the answer to appear anywhere within a sentence. this method is compatible with any type of model (whether clm or mlm) and any tokenization. by evaluating the log-likelihood score of the entire statement instead of just its continuation or the single answer token, lm-pub-quiz overcomes the limitations of traditional [mask]predict approaches @xcite without relying on text-continuation capabilities. ## conclusion and outlook in this paper, we presented lm-pub-quiz, an easy-to-use and versatile open source library for knowledge probing that can be seamlessly used with the bear probe. the framework covers two important use cases: monitoring knowledge during continual pre-training (and domain adaptation) and analyzing existing pre-trained lms. we are actively working on extending the leaderboard and strongly encourage the community to participate. we will continue development on the library to support further use cases and welcome any input, whether in the form of feedback or contributions to the code base. we are working to extend the bear probe to additional knowledge bases in order to expand on the domains of knowledge that can be evaluated with lm-pub-quiz.