Measuring temporal effects of agent knowledge by date-controlled tool use (2025)

Abstract: Temporal progression is an integral part of knowledge accumulation and update. Web search is frequently adopted as the grounding for agent knowledge, yet an improper configuration affects the quality of the agent’s responses. Here, we assess the agent behavior using distinct date-controlled tools (DCTs) as a stress test to measure the knowledge variability of large language model (LLM) agents. We demonstrate the temporal effects of an LLM agent as a writing assistant, which uses web search to complete scientific publication abstracts. We show that the temporality of search engines translates into tool-dependent agent performance but can be alleviated with base model choice and explicit reasoning instructions such as chain-of-thought prompting. Our results indicate that agent design and evaluations should take a dynamical view and implement effective measures to account for the temporal influence of external resources to improve agent reliability.

## introduction ai agents based on llms and equipped with tools @xcite are wellsuited for complex real-world tasks @xcite because of their extended capabilities. their potential to become virtual assistants, paraprofessionals, or "copilots" holds promise for improving the productivity and creativity of the scientific, medical workforces and beyond @xcite @xcite . the evaluation standards for ai agents are still in flux @xcite and they are urgently needed in specialized domains and realistic scenarios where the outcomes convey greater bearing on their adoption. recent works demonstrated the feasibility of llms in predicting temporal events @xcite and carrying out time series forecasting @xcite , but their equivalents in agentic systems are not yet realized. scientific knowledge and claims have a strong temporal dependence but they have so far been less studied in the context of generative language models @xcite . we devised a text completion task as a proxy to measure the agent's usability as a writing assistant with access to external sources (see fig. 1a ). web search is an essential tool for grounding agent knowledge in the current and bygone worlds @xcite ) and it appears in many applications as a capability extender for models @xcite . nevertheless, web search is subject to the recency and primacy bias of the search engine @xcite and the cognitive bias of the users who seek and collect information @xcite . the term search engine manipulation effect @xcite was coined to refer to the search results' influence on public opinions of societal issues. independent of search engines, factual and scientific knowledge also experience constant but necessary updates over time @xcite . while temporal generalization remains challenging for language models @xcite , explicit tuning of time-related tool parameters in llm agents can offer an alternative way to reduce temporal effects of the base model. these effects are a source of performance reliability issues of agentic systems that warrant investigation @xcite . date control in reality can manifest passively because the toolinterfaced computer programs have an intrinsic time stamp or a versioned release over time @xcite . alternatively, date control can be imposed actively because of copyright, paywall, or local policy. content access in the past can be controlled retroactively as policy changes (aral and dhillon, 2021). from a technical standpoint, invoking different dcts is equivalent to changing the environment (here means the surface web, see fig. 1a ) of the agent, which requires the agent to adjust to in task execution. stress testing is the ultimate test for model behavior and trustworthiness. in the time domain, out-of-sample (oos) testing is typically used for temporal prediction methods @xcite . analogous oos assessments in the text domain include predicting future events @xcite or generating hypotheses @xcite conditioned on existing (e.g. past) knowledge. we investigate the comparable problem from the tool use perspective, where the agent has access to changing internet-scale information. because scientific breakthroughs often lead to significant knowledge updates, they are good markers for temporal knowledge progression 2 . in this work, we aim to investigate the following research questions: rq1: can we manipulate agent knowledge by imposing date restrictions on the tools? rq2: can agents determine the optimal datecontrolled version of a tool to use for a task? our contributions along these directions are: (i) formulation of a tool-based stress test for timedependent knowledge for llm agents; (ii) introduction of the scibreak dataset containing the publication records of public-endorsed scientific breakthroughs from 2000 to 2024. (iii) investigation of the temporal effects of llm agent performance and behavior. besides, we also discuss the impact of temporal information on the agent capability and 2 although the judgement on breakthroughs is ultimately subjective and can change with time, our motivation to use them is because of their noticeable footprints on the internet. usability and its implications. out-of-sample testing classical and learningbased time series forecasting commonly employ temporal oos performance tests @xcite @xcite to ensure model credibility and usability. it is also relevant from an online learning perspective where data are streamed in sequentially and are subject to distribution shifts. in deep learning, oos testing is used to provide risk-based selfcertification for neural networks @xcite . in generative models, it has been used for prompt selection @xcite and controlled generation @xcite in language models and for quality assessment of synthetic signal generators @xcite . the tool t converts the source information s from the environment into the observation o to support agent reasoning and action. in fig. 1b , s refers to the surface web and o the ranked snippets. a web-search agent has an implicit parameter @xmath0.e. the current date) for the tool t , but it can be modified to an arbitrary value @xmath1, which changes the observation in eq. ( 2 ). definition 3.3 (tool-based stress test). a performance test that induces stress conditions by adjusting the tool parameters of an agentic system. a temporal version of the test alters the time information of tools and therefore measures the reliability of agent performance under such conditions. ## testing framework implementation dataset we constructed the scibreak dataset, which has a clear time-delimited footprint on the internet-scientific breakthroughs. we extended the dataset collated in @xcite to the year of 2024. each year contains up to ∼ 20 publications, including multiple publications contributing to one breakthrough. ## discussion agent vulnerability and tool-based control our work shows that agents with access to external tools are subject to manipulation by corrupted tools @xcite ) to compromise their generated information for knowledge-intensive domains, extending the previous example on misinformation in llms @xcite . we provide evidence that agentic reasoning and model capabilities can counter the limited information quality of search engines. in agentic search, carefully designed controls will allow filtering of unreliable information and improve agent performance. imposing a date restriction on search is similar to reranking and partial deletion of the search results. therefore, agent designs with verification of content freshness and temporality will ensure more reliable use. robustness and reproducibility of agentic systems agentic systems for scientific problems should adapt to different levels of prior knowledge available to the domain @xcite . from the robustness viewpoint, temporal shifts can be counteracted through the use of external resources. task-oriented requirements specification @xcite is useful for improving the usability and avoiding unnecessary artifacts from model imperfections and the reliability of external tooling and information sources. from the reproducibility viewpoint, agentic tool use should always incorporate essential information of the key parameters. our work indicates that more research is needed in principled maintenance of agentic frameworks under constant updates of external resources to facilitate reliable agent design @xcite .