Updating Large Language Models’ Memories with Time Constraints (2024)

Abstract: By incorporating the latest external knowledge, large language models (LLMs) can modify their internal memory. However, in practical applications, LLMs may encounter outdated information, necessitating the filtering of such data and updating of knowledge beyond internal memory. This paper explores whether LLMs can selectively update their memories based on the time constraints between internal memory and external knowledge. We evaluate existing LLMs using three types of data that exhibit different time constraints. Our experimental results reveal the challenges most LLMs face with time-constrained knowledge and highlight the differences in how various LLMs handle such information. Additionally, to address the difficulties LLMs encounter in understanding time constraints, we propose a two-stage decoupling framework that separates the identification and computation of time constraint into a symbolic system. Experimental results demonstrate that the proposed framework yields an improvement of over 60% in ChatGPT’s performance, and achieves a 12-24% enhancement in state-of-the-art LLM GPT-4.

## introduction large language models (llms) can provide answers that differ from their internal knowledge when supplied with external information @xcite . this capability facilitates the updating of incorrect or outdated information within llms @xcite . for instance, as illustrated in figure 1 (a), when asked "who is the prime minister of the uk?", the llm can provide the updated answer "rishi sunak," based on the latest knowledge, instead of relying on its outdated memory like "boris johnson." in practical applications such as retrievalaugmented tasks @xcite , outdated information may also be inputted into llms alongside the latest knowledge. this necessitates llms to eliminate knowledge overlapping with their memory and update their internal memory with new information. a straightforward approach involves selectively updating the internal memory of llms based on the time constraint relationship between the question, internal memory, and the external knowledge. for example in figure 1 (b), given the question "how many la metro rail stations as of 2024", the llm's original internal memory states "as of 2021, there are 93." the first given piece of knowledge has a time constraint of "may 20, 2016", which is covered by the llm's current internal memory range, thus no update is needed. the second piece of knowledge has a time constraint of "2022-2024", which extends beyond the llm's internal memory and necessitates an update. ignoring these time constraints can lead llms to provide answers that either exceed the scope of the intended question constraints or result in redundant updates to already covered knowledge. this paper conducts a comprehensive and controlled study of how llms understand and update time-constrained knowledge. specifically, we construct test data based on three time constraint relations between internal memory and external knowledge (i.e., complete inclusion, intersection, and disjunction) to examine whether llms can comprehend and selectively update time-constrained knowledge. additionally, we investigate the sensitivity of llms to the sequential order of timeconstrained knowledge and the role of internal memory in the process of knowledge updating. our experiments reveal several key findings: (1) most llms struggle to update memory with time-constraint. except for gpt4, the other llms (e.g., llama2, mistral) exhibit poor abilities for updating memory with time constraints. additionally, chatgpt performs worse in this regard compared to smaller open-source models across multiple scenarios. in contrast to the poor temporal reasoning of llms @xcite , the discovery of the the relationship of time constraints are easily achievable in symbolic systems @xcite . for example, in a python solver, representing time-related information using specific data structures enables checking for temporal contradictions among multiple pieces of information. on the other hand, llms have shown strong capabilities in information extraction @xcite . based on this, we propose a two-stage decoupling framework for time-constrained knowledge updating. we limit llms to information extraction only and delegate information reasoning to symbolic systems (i.e., python solver). we conduct experiments on mistral, chatgpt, and gpt4, and the results demonstrate significant performance improvements for these three llms. ## dataset construction we introduce a framework to construct timeconstrained knowledge. following prior work @xcite @xcite , we adopt question answering (qa) task as the testbed for evaluation. first, we collect questions whose answers may change over time. second, we elicit llms' internal memories about these questions. finally, we construct new timeconstrained knowledge that need to be updated into llms' memories. ## two-stage decoupling framework the above experimental results indicate that, except for gpt-4, most llms exhibit weak abilities to identify and update time-constrained knowledge. one possible reason is that llms lack of understanding of temporal reasoning @xcite . in contrast, llms demonstrate excellent performance in information extraction @xcite . additionally, most temporal reasoning problems can be solved through symbolic computation @xcite , such as a python solver. therefore, we propose a two-stage framework for updating time-constrained knowledge. we decouple information extraction and temporal reasoning into two separate modules. specifically, this framework includes an evidence extractor for extracting time-sensitive information and a timeline builder for calculating the final answer based on the extracted information. the overall architecture is illustrated in figure 4 . ## conclusion in this paper, we explore the capabilities of llms for updating knowledge with time constraints. we propose a framework for constructing test data for time-constrained knowledge and construct test data to evaluate llms including llama2, vicuna, mistral, chatgpt, gpt4, respectively. experimental results reveal that most llms lack the ability to update time-constrained knowledge. on the other hand, different models also exhibit different decision styles for time-constrained knowledge. additionally, we find that time constraints in question is a trigger that can change the llms' decision styles. finally, we propose a two-stage decoupling framework that decouples the discovery and computation of time constraints from llms into a symbolic system. experimental results show that this decoupling approach can bring chatgpt an improvement of over 60%, and also has a 12-24% improvement for state-of-the-art llms gpt4.