E pi K -Eval: Evaluation for Language Models as Epistemic Models (2023)

Abstract: In the age of artificial intelligence, the role of large language models (LLMs) is becoming increasingly central. Despite their growing prevalence, their capacity to consolidate knowledge from different training documents—a crucial ability in numerous applications—remains unexplored. This paper presents the first study examining the capability of LLMs to effectively combine such information within their parameter space. We introduce EpiK-Eval, a novel question-answering benchmark tailored to evaluate LLMs’ proficiency in formulating a coherent and consistent knowledge representation from segmented narratives. Evaluations across various LLMs reveal significant weaknesses in this domain. We contend that these shortcomings stem from the intrinsic nature of prevailing training objectives. Consequently, we advocate for refining the approach towards knowledge consolidation, as it harbors the potential to dramatically improve their overall effectiveness and performance. The findings from this study offer insights for developing more robust and reliable LLMs. Our code and benchmark are available at https://github.com/chandar-lab/EpiK-Eval

## introduction developing systems that can reason through language understanding has been a cornerstone in natural language processing research. recent progress @xcite @xcite has showcased notable advancements in a variety of reasoning tasks @xcite @xcite @xcite . arguably, the ability of lms to act as knowledge bases @xcite has been a large factor in these successes. however, observed errors @xcite on tasks which entail learning dependencies among multiple facts can be potentially 10% 45% 45% tom ate an apple. tom ate a pear. bob ate an orange. tom ate an apple. tom ate a pear. tom ate an orange. ## epistemology & language models epistemic frameworks @xcite are formal systems used to represent knowledge, belief and the uncertainty that entails what a reasoning system knows and/or believes. this is enabled through organizing the knowledge observed by the system. the rules to combine the knowledge in the abstract framework governs combining a new information to the current set of information, or when to ignore the new information, and using the current beliefs to anticipate related events. while lms behave as kbs to store known relations, epistemic logic provides us with the inspiration to describe how these models organize and update their knowledge. consider the example from figure 1 , where we have the knowledge x 1 : "tom ate an apple.", x 2 : "tom ate a pear." and x 3 : "bob ate an orange.". prompted with the question "what did tom eat?", the model must recall knowledge from within its parameter space. it has to connect x 1 and x 2 while also ignoring x 3 . to answer the query, a system is expected to consolidate the information and retain a knowledge state over the information it had seen until then. however, an inability to draw the connections would leave the facts disconnected. we describe the model that struggles to consolidate as type i, and one that is better at it and infer over a consolidated knowledge state as type ii. with lms being used in real-world scenarios where information is frequently presented as a pe-riodic flow, it is necessary that they use such information appropriately during inference. while techniques like self-prompting and generation over selfretrieval are gaining popularity, the performance relies on the quality of the prompt, which adds to the robustness concerns on the performance of lms on varying reasoning tasks. inspired by epistemology, we design epik-eval to diagnose whether lms comply with a first-order knowledge state following a sequence of facts which holds a consolidated summary of information during inference. ## epik-eval the epik-eval benchmark presents a suite of novel, narrative-based diagnostic tasks, meticulously designed to evaluate a lm's capacity to construct a comprehensive, unified knowledge state. dataset: our benchmark comprises 18 tasks, which are questions about relations between facts and events in stories, e.g., "does x happen before/after y?". ranking tests understanding of relative amounts, frequency, and ranking. temporal tests if the model has learned temporal dependencies in the data, such as what events follow each other. causal tests understanding of cause-effect. uniqueness tests understanding of exclusivity or uniqueness in the data. ## experiments we experiment with three different llms: t5 (raffel et al., 2020), its instruction-tuned variant flan-t5 @xcite , and opt @xcite . for t5 and flan-t5 models, we benchmark sizes from small to xl. for the opt model, we benchmark sizes 125m, 350m, 1.3b and 2.7b parameters. unless otherwise stated, the reported performance is on the test set. performance scores presented in this section are always averaged over the 18 tasks of our benchmark. individual task performance can be found in appendix b, and training details are provided in appendix a. ## discussion consolidating knowledge in language models: our study delineates the limitations of language models in consolidating knowledge across different text sequences, compared to a noticeably stronger performance when working within a single text sequence. we attribute this disparity primarily to the core objective of such models: to enhance word prediction within given sequences, while also using knowledge from previously processed text sequences, encoded in the model's parameters. current pre-training objectives such as masked and causal language modeling @xcite potentially prioritize learning dependencies within text sequences over those spanning across multiple ones. for instance, a cause-and-effect relationship could exist between two sequences. however, if the content of the first does not explicitly help in predicting the second's content, the model might not learn this relation. consequently, numerous inter-sequence dependencies in the training corpus, which may hold significant importance in downstream tasks, may be ignored owing to their perceived irrelevance in the next-word prediction task. in contrast, the model can readily establish correlational dependencies within individual sequences which can even lead to the direct memorization of text, a frequent occurrence in llms @xcite @xcite . in light of these arguments and results, we assert the need to revisit the training objectives of language models. to utilize these models effectively, we should prioritize devising training methods that capture and consolidate the numerous information dependencies within the training corpus. a potential avenue to explore could be to guide these models in consolidating their knowledge via methods such as rl-hf @xcite or self-taught @xcite . ## conclusion in this paper, we presented the epik-eval benchmark, a tool designed specifically to evaluate the proficiency of lms in consolidating their knowledge for problem-solving tasks. our findings underscore the limitations of current lms, which appear to mostly maintain a disjoint knowledge state of training observations. further, we note a significant performance gap and an increased rate of hallucinations for models trained on segmented narratives compared to those trained on unsegmented ones. we attribute these discrepancies to the training objectives of the models, which underscores the need to more effectively model the dependencies within the training corpus. by highlighting current limitations and opportunities for improving lms, these results delineate paths for future research, hopefully enabling the growth of language models beyond simple knowledge bases.