D y K now: Dynamically Verifying Time-Sensitive Factual Knowledge in LLM s (2024)

Abstract: LLMs acquire knowledge from massive data snapshots collected at different timestamps. Their knowledge is then commonly evaluated using static benchmarks. However, factual knowledge is generally subject to time-sensitive changes, and static benchmarks cannot address those cases. We present an approach to dynamically evaluate the knowledge in LLMs and their time-sensitiveness against Wikidata, a publicly available up-to-date knowledge graph. We evaluate the time-sensitive knowledge in twenty-four private and open-source LLMs, as well as the effectiveness of four editing methods in updating the outdated facts. Our results show that 1) outdatedness is a critical problem across state-of-the-art LLMs; 2) LLMs output inconsistent answers when prompted with slight variations of the question prompt; and 3) the performance of the state-of-the-art knowledge editing algorithms is very limited, as they can not reduce the cases of outdatedness and output inconsistency.

## introduction large language models (llms) have been compared to traditional knowledge repositories, such as knowledge bases, knowledge graphs, and search engines regarding their capability to retrieve factual knowledge @xcite @xcite . a critical requirement for a reliable knowledge repository is to maintain the accuracy of the factual information it contains. the factual knowledge has a dynamic nature and can change significantly from what has been first inserted; and in the case of llms what was observed during the training stage. llms are static models that are prone to generating invalid and contradicting information, and eventually getting outdated over time. they derive their knowledge from vast and often unoptimized collections of data snapshots, that are collected at different timestamps @xcite , and typ-"what is cristiano ronaldo's club? let's ask llms!" ## dyknow : dynamic knowledge validation the benchmark for evaluating time-sensitive knowledge in llms must be model-agnostic and long-lasting since it must not become outdated as the models. a static benchmark lacks these characteristics, cannot capture the changing world, and can lead to data contamination. we present a cost-effective approach to dynamically benchmark the factual knowledge in llms using wikidata. wikidata is a multilingual knowledge graph that is continuously and collaboratively edited to maintain up-to-date information @xcite . factual knowledge in wikidata is presented by properties that connect subject nodes to attribute values. for example, "cristiano ronaldo's current football club" factual knowledge is presented by the property "member of sports team" that connects the subject "cristiano ronaldo" to the current attribute value, at the time of paper "al-nassr". furthermore, the attribute values in wikidata are accompanied by qualifiers, which provide additional context and specificity regarding the attribute values such as geographical locations, measurement units, as well as start and end dates for attribute values that have a temporal validity interval. for instance, the attribute value "al-nassr" is accompanied by start and end date quantifiers "2023-now". besides the current attribute value for each factual triplet, wikidata maintains all the previously correct attribute values in addition to their corresponding start and end date quantifiers, indicating the corresponding temporal validity intervals. therefore, the complete list of attributes for "cristiano ronaldo's current football club" factual knowledge consists of [al-nassr 2023-now , manchester united f.c. 2021-2022 , juventus fc 2018-2021 , real madrid 2009-2018 ,...] (figure 1 ). instead of relying on static ground truth values, we evaluate the models' outputs with the list of attribute values retrieved dynamically from the wikidata knowledge base at the time of evaluation. we assess the knowledge of the model regarding each fact as: ## validating dyknow to assess the efficacy of the proposed approach, we evaluate 24 llms on 130 time-sensitive facts including countries' politicians, athletes' clubs, and organizations' roles. this allow us to introduce diversity in the dataset by having human subjects, organization subjects, and country subjects with a diverse set of properties to query the models. ## discussion in this section, we discuss our findings in relation to the introduced research, as well as avenues for future work. ## conclusion we have investigated the process of keeping llms' knowledge up-to-date and presented an approach to dynamically benchmarking this knowledge via wikidata. dynamic benchmarks are a promising solution to address the known limitations of static benchmarks, such as outdatedness and data contamination. our results indicate that llms differ from traditional knowledge repositories, making it important to investigate what types of knowledge these models can reliably manage and what types of querying and alignment operations they support. we encourage further community engagement to expand dyknow into a current and active benchmark. benchmark. the performance of the editing methods on the ripple effect of edited time-sensitive facts is not evaluated in this work. the evaluated editing methods are focused on updating the llms and do not consider the other operations of removing the knowledge from the model or adding knowledge to the llm. lastly, the evaluations of editing methods are limited due to a lack of computation resources, as we could not experiment with larger open-source models.