BEAR : A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models (2024)

Abstract: Knowledge probing assesses to which degree a language model (LM) has successfully learned relational knowledge during pre-training. Probing is an inexpensive way to compare LMs of different sizes and training configurations. However, previous approaches rely on the objective function used in pre-training LMs and are thus applicable only to masked or causal LMs. As a result, comparing different types of LMs becomes impossible. To address this, we propose an approach that uses an LM’s inherent ability to estimate the log-likelihood of any given textual statement. We carefully design an evaluation dataset of 7,731 instances (40,916 in a larger variant) from which we produce alternative statements for each relational fact, one of which is correct. We then evaluate whether an LM correctly assigns the highest log-likelihood to the correct statement. Our experimental evaluation of 22 common LMs shows that our proposed framework, BEAR, can effectively probe for knowledge across different LM types. We release the BEAR datasets and an open-source framework that implements the probing approach to the research community to facilitate the evaluation and development of LMs.

## introduction pre-trained language models (lms) are the backbone of current state-of-the-art nlp approaches. a key property is the syntactic and semantic knowledge stored in their internal parameters, allowing them to generalize beyond given training data when fine-tuning for a specific downstream nlp task. due to their importance and the large number of proposed lms, prior work has sought to improve the ability to measure the amount of factual knowledge encoded in lms, thereby facilitating the comparison of different lms @xcite ; po- ## lama and follow-up work we discuss the technical details of the lama probe first, followed by an analysis of its weaknesses. lama evaluation data. the lama benchmark was originally composed of four separate datasets named after their respective sources: squad @xcite , googlere foot_2 , conceptnet @xcite , and t-rex @xcite . however, subsequent research mainly focused exclusively on the t-rex subset. its knowledge base comprises 41 relations derived from wikidata. each relation contains at most 1,000 relation instances in the form of subject-relationobject triples ⟨s, r, o⟩ where s is the subject (e.g., "france"), r the relation (e.g., has-capital), and o an object (e.g. "paris"). there are three types of relations in lama: 1-1 (one-to-one, e.g., has-capital), n-1 (many-toone, e.g., has-language), and n-m (many-tomany, e.g., shares-border-with). relations of the n-1 type allow multiple subjects to relate to one object, while the latter permits many subjects to be associated with numerous objects. ## bear probe we base our evaluation on using an lm's inherent ability to estimate the log-likelihood of a given sentence. our main idea is to restrict the space of possible answers for each relation instance and create a set of options that are then ranked by their log-likelihood values. ## bear dataset our proposed probing approach requires a dataset with a restricted answer space. following the analysis in section 2 we additionally desire (1) the answer space to be balanced, (2) a single correct answer per relation instance, (3) a balanced number of instances per relation, and (4) a focus on knowledge that could reasonably be found in corpora other than wikipedia. ## experiments we present an experimental evaluation using the bear probe on a selection of lms, compare our results to earlier probes, and discuss the results. compared lms. we compare a total of 22 lms, as listed in and opt families, along with newer models such as llama2, gemini, and mistral. we assess 5 different sizes for the opt models to examine the relationship between the bear score and increasing model sizes. bear score. we compute the bear score for each of the three template options per relation individually and report the average across templates as well as the standard deviation. ## conclusion we presented bear, a relational knowledge probe applicable to both causal and masked lms. since our proposed approach imposes no restrictions on the evaluation data, we created a large evaluation dataset that addresses issues of answer skews, domain and template bias, and the correctness of facts identified by ourselves and prior work. we publicly release bear for use by the research community.