G row OVER : How Can LLM s Adapt to Growing Real-World Knowledge? (2024)

Abstract: In the real world, knowledge is constantly evolving, which can render existing knowledge-based datasets outdated. This unreliability highlights the critical need for continuous updates to ensure both accuracy and relevance in knowledge-intensive tasks. To address this, we propose GrowOVER-QA and GrowOVER-Dialogue, dynamic open-domain QA and dialogue benchmarks that undergo a continuous cycle of updates, keeping pace with the rapid evolution of knowledge. Our research indicates that retrieval-augmented language models (RaLMs) struggle with knowledge that has not been trained on or recently updated. Consequently, we introduce a novel retrieval-interactive language model framework, where the language model evaluates and reflects on its answers for further re-retrieval. Our exhaustive experiments demonstrate that our training-free framework significantly improves upon existing methods, performing comparably to or even surpassing continuously trained language models.

## introduction in natural language research, many knowledgeintensive tasks have been actively studied, including open-domain qa @xcite @xcite , factchecking @xcite , entity linking @xcite , and open-domain dialogue @xcite , to name a few @xcite . such a knowledge-intensive task is mostly to utilize world knowledge to generate a proper answer for a given query @xcite . however, the amount of real-world knowledge is often too enormous for models to fully store them in the parameters. thus, in most scenarios, a retriever is employed to seek relevant para-messi's free kicks are a thing of beauty. how does he rank among other players in that aspect? that's true. as of september 2023, messi ranks 5 th all time in goals scored from direct free kicks with 65! ## related work temporal sensitivity. temporal misalignment occurs when training and test datasets originate from different time periods. past studies @xcite report poor performance in downstream tasks when making predictions beyond the training period. thus, recent research @xcite @xcite @xcite efforts to evaluate how llms handle time-sensitive information. in particular, there have been approaches to utilize a retriever for time-sensitive knowledge. @xcite and @xcite report that, even with an updated evidence corpus, language models trained on previous data struggle to respond to questions in the present. however, @xcite show that llms can adjust their generated responses to recently retrieved documents provided by prompting. still, when failing to retrieve appropriate documents, llms may produce outdated answers. to address this issue, we propose a retrieval-interactive llm framework that allows the llm to provide feedback to the retriever to fetch more relevant documents when the answer is less reliable. ## the growover dataset growover comprises two distinct datasets: qa and dialogue. growover-qa is designed to evaluate the ability to recall entities, while growover-dialogue features userexpert interactions over 3-4 turns to highlight generation capabilities. each instance from both datasets is annotated with the evidence text and the type: unchanged, changed, or new. our goal is to evaluate the retention of unchanged knowledge, the updating of changed knowledge, and the acquisition of new knowledge, aligning with the objectives of ckl. article selection. growover is based on wikipedia snapshots1, which contain a vast amount of world knowledge. although there is no limitation to applying our algorithm to generate qa and conversation for any articles, we select the articles linked to portal:current events wikipedia article from january 2023 to december 2023 (about 12k articles). overall process. we create initial qa and dialogue instances using the 2023-08-20 wikipedia snapshot using gpt-4. for each subsequent snapshot, we label each sentence in articles as unchanged, changed, or new by comparing it to the previous month's snapshot. then, we retain qa and dialogue instances when the evidence text is labeled as unchanged, and create new instances from new or changed sentences. this process is repeated for each month's new snapshot from september through december. algorithms and prompt templates for sentence labeling and data generation are detailed in appendix c. also, the statistics of datasets are provided in appendix e. ## approach if the llm has been trained on outdated data or has never been trained on new data, it may not be able to answer the questions on new information correctly. however, updating the parameters of such models should be conducted with the greatest caution to avoid potential side effects, such as catastrophic forgetting. therefore, our approach ensures that neither the llm nor the retriever is continuously trained with new data. to adapt llms to rapidly evolving world knowledge, we propose rilm, as shown in figure 3 . while freezing both the llm and the retriever, we introduce the decision gate, which decides whether to accept the llm's answer based on its certainty score (section 4.1). if the answer is not sufficiently confident, the retrieval-generation process is performed again, termed adaptive re-retrieval (section 4.2). in this process, the llm's previous output is fed back to the retriever to fetch documents again, enabling the llm to generate better answers. similar to our approach, @xcite propose using reflection tokens to confirm output relevance, support, or completeness. however, their method requires training llms to predict reflection tokens. in contrast, rilm only involves training the certainty classifier. additionally, for long-form text generation, @xcite generate the next sentence and then concatenate it with the question to form a retrieval query. they exclude tokens generated with low probability to avoid interrupting retrieval. our approach also uses generated answers for re-retrieval but considers all generated tokens only to the extent that the llm is certain about the answer. ## conclusion to evaluate whether llms can adapt to the fast-evolving world knowledge, we propose growover-qa and growover-dialogue. our benchmarks annotate the evidence text and introduce dialogue tasks to evaluate retriever-augmented ralm comprehensively. furthermore, we suggest rilm, an interactive retriever-generator framework by simply training a classifier for llm to predict reliability itself. through rigorous experiments, we show that our method can be on par with or surpass continuously pre-trained llms even without pre-training. however, even with retrieval, we observe that the performance degrades over time. thus, we hope our benchmarks can be valuable resources to detect when to update the retriever or llm in future work. additionally, we anticipate further research into optimizing the use of retrievers to reduce the frequency of model updates.