From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts (2025)

Abstract: Sample efficiency is a crucial property of language models with practical implications for training efficiency. In real-world text, information follows a long-tailed distribution. Yet, we expect models to learn and recall frequent and infrequent facts. Sample efficient models are better equipped to handle this challenge of learning and retaining rare information without requiring excessive exposure. This study analyzes multiple models of varying architectures and sizes, all trained on the same pre-training data. By annotating relational facts with their frequencies in the training corpus, we examine how model performance varies with fact frequency. Our findings show that most models perform similarly on high-frequency facts but differ notably on low-frequency facts. This analysis provides new insights into the relationship between model architecture, size, and factual learning efficiency.

## introduction with the continued advancement of language models (lms), comparing different architectures across various tasks and evaluating their performance using appropriate metrics becomes increasingly essential. these comparisons offer valuable insights into each architecture's general strengths and limitations. sample efficiency is a key property of lms, as sample-efficient models require less training and are thus more cost-effective @xcite . as the lm processes large text corpora during pretraining, we are interested in assessing how efficiently each model learns specific relational facts comprising a subject, relation, and object. a core question in this context is how different architectures handle the challenge of learning and retaining rare versus frequent facts. if two models are trained on the same dataset, their sample efficiency can be assessed by determining how often a fact must appear before each model successfully learns it @xcite , 2023). models that rely predominantly on frequent facts while struggling with rarer ones-an issue caused by the long-tailed distribution of information in natural text @xcite -are considered sample-inefficient. conversely, sampleefficient models should achieve higher accuracy on rare facts while maintaining strong performance on more common ones. to assess a model's factual knowledge, we use the bear probe @xcite , which evaluates the model's ability to recall factual information across a wide range of subject-relation-object triples. an lm's factual knowledge can be probed by passing statements into the model (e.g., "the capital of germany is ...") and evaluating its output to determine the represented knowledge of an lm @xcite @xcite . bear enables evaluation of both causal and masked lms by constructing multiple answer choices, where each instance is transformed into a set of natural language statements: one for each answer option (e.g., "berlin", "paris", "buenos aires", etc. for the relation has-capital and the subject "germany"). the lm assigns loglikelihood scores to these statements, which are then ranked to determine the predicted answer. since bear contains no information about the pre-training data, it alone cannot be used to assess the sample efficiency.to address this, we need to not only determine whether the lm can correctly recall a given fact but also how many times it encountered it during pre-training (in the following, we call these "frequencies"). to create a correct sample efficiency evaluation procedure, we require an approach to estimate frequencies of facts from bear within a text corpus used for pre-training (see figure 1 ). for this study, we employ a simple matching-based heuristic (see section 3.1). though unable to capture every occurrence of a fact, we assume it to be sufficiently accurate to predict the relative frequencies. given the information about how often an lm has encountered specific facts and whether it can recall them correctly, we must determine how to translate these fact-level data to a sample efficiency measure. rather than estimating the point at which an lm transitions from not knowing to having learned the fact, we propose a more nuanced perspective: measuring the incremental gain in factual knowledge as a function of the number of training samples. to operationalize this, we introduce two complementary metrics, which we use to quantify and compare the sample efficiency of different models over varying levels of fact exposure. ## approach to evaluate a model's sample efficiency, we employ a three-step approach. we build on bear and extend the probe by collecting fact frequencies (see section 3.1) for a given pre-training corpus. we then train several lms on this corpus (section 3.2). this way, we can estimate how often a model has encountered a specific fact during its pre-training (and at which point). in section 3.3, we introduce two novel sample efficiency metrics which produce aggregated scores based on the model's response to each sample and the sample's frequency. ## empirical evaluation leveraging the proposed approach allows us to address the following questions: (1) which model architecture demonstrates higher levels of sample efficiency, (2) and how well a model recalls facts throughout the training. ## conclusion we presented a sample efficiency evaluation framework that compares lms' ability to learn facts given a text corpus and the bear probe. the framework consists of a fact-matching algorithm that extracts fact frequency statistics from a sizable data set and two sample efficiency metrics. we trained several state-of-the-art lms in a controlled setting, ensuring the validity of the evaluation, and provided a detailed analysis of the different architecture results. the performance on high-frequency facts indicates less divergence between models regarding size. in contrast, performance on low-frequency facts demonstrates the increased sample efficiency gained with model size. the proposed metrics are capable of identifying the superiority in sampleefficiency of the transformer-based llama models, achieving the highest scores in all metrics, with the state-space-based mamba2 models closing behind. the proposed metrics correlate strongly in respect to the final model stages as well as across the training. this indicates that a different property is measured than in raw accuracy. additional experiments show, that the metrics are relatively robust to varying fact frequency distributions in pre-training data. we believe the plausibility of the design choices together with these findings make the metrics strong candidates for measuring sample efficiency.