Can Language Models Serve as Temporal Knowledge Bases? (2022)

Abstract: Recent progress regarding the use of language models (LMs) as knowledge bases (KBs) has shown that language models can act as structured knowledge bases for storing relational facts. However, most existing works only considered the LM-as-KB paradigm in a static setting, which ignores the analysis of temporal dynamics of world knowledge. Furthermore, a basic function of KBs, i.e., the ability to store conflicting information (i.e., 1-N, N-1, and N-M relations), is underexplored. In this paper, we formulate two practical requirements for treating LMs as temporal KBs: (i) The capacity to store temporally-scoped knowledge that contains conflicting information and (ii) the ability to use stored knowledge for temporally-scoped knowledge queries. We introduce a new dataset called LAMA-TK which is aimed at probing temporally-scoped knowledge, and investigate the two above requirements to explore the LM-as-KB paradigm in the temporal domain. On the one hand, experiments show that LMs can memorize millions of temporally-scoped facts with relatively high accuracy and transfer stored knowledge to temporal knowledge queries, thereby expanding the LM-as-KB paradigm to the temporal domain. On the other hand, we show that memorizing conflicting information, which has been neglected by previous works, is still challenging for LMs and hinders the memorization of other unrelated one-to-one relationships.

## introduction recently, language models (lms) such as bert @xcite and t5 @xcite have been suggested as an alternative to world knowledge bases (kbs) @xcite . the parameters of these models appear to store extensive real-world knowledge during training and stored knowledge can be recalled by filling cloze statements (e.g. "dani alves plays with @xcite . -> barcelona"). as a result, recent works have considered lms for tasks such as closed-book question answering @xcite , automated fact-checking @xcite , and knowledgegrounded dialogue systems @xcite . relational facts in world knowledge often change with time. for example, "michael jordan played for washington wizards." is true only from 2001 to 2003. however, most existing works only considered the lm-as-kb paradigm in a "static" setting, ignoring the temporal dynamics of world knowledge. however, this temporallyscoped knowledge raises several potential challenges for the lm-as-kb paradigm. conflicting information while training on the large textual corpus, the model will inevitably encounter conflicting information (i.e., 1-n, n-1, n-m relations), e.g., "giannis antetokounmpo played for filathlitikos / milwaukee bucks". @xcite limits the time period of facts to reduce the amount of conflicting information. however, conflicting information still exists, from the players who played for a team to the politician who held multiple positions. these conflicting facts will hinder the memorizing process and cause the model to have difficulty memorizing all correct answers. correlation between temporal scopes temporal facts usually contain temporal scopes (e.g., a start time and an end time), and a strong correlation is present between these timestamps. for example, "shinzo abe was the prime minister of japan from 2006 to 2007." and "shinzo abe was the prime minister of japan from 2012 to 2020." are two temporally-scoped facts. these facts have the same subject, object, and predicate but different temporal scopes. as temporal knowledge bases, lms need to memorize not only the timestamps associated with the facts but also the matching relationships between temporal scopes. implicit temporal knowledge temporallyscoped facts usually contain implicit facts. for example, "françois hollande served as president of the french republic from 2012 to 2017" contains the following facts: "françois hollande served as president of the french republic in 2015" and "françois hollande was elected president of the french republic in 2012". these implicit facts are not directly mentioned in temporally-scoped facts. temporally-scoped knowledge widely exists in real-world knowledge bases like wikidata. however, existing qa datasets such as lama @xcite , natural questions @xcite focus on probing static knowledge, ignoring the temporal dynamics of world knowledge. the temporal dataset templama @xcite focuses on querying factual objects with single timestamps, ignoring the temporally-scoped information such as the start and end times. moreover, temporal facts often contain extensive conflicting information, but previous works did not pay enough attention to these conflicts. they explored lm-as-kb within 1-1 relations (e.g. born in) or discarded facts with multiple objects. therefore, we propose lama-tk (short for language model analysis for temporal knowledge), a new dataset for probing lms for temporally-scoped knowledge. lama-tk queries temporal knowledge including entity names and special timestamps, and reserves all correct answers for each query. based on lama-tk, we introduce two practical questions for lms as temporal kbs to explore the lm-as-kb paradigm in the temporal @xcite @xcite @xcite domain. we examine lms on two basic functions of kbs: the storage capacity and the use of stored temporal knowledge, and identify the challenges mentioned above during the experiments (section 4). first question: what is the storage capacity of lms for storing temporal knowledge? here, we ask the models to memorize millions of temporal facts and record the storage performance of these lms (section 4.1). results show that lms can memorize millions of temporal facts with relatively high accuracy. however, we also show that conflicting information poses a great challenge to the storage capacity of lms and hinders the memorizing process of other unrelated facts. second question: can lms use stored temporal knowledge for temporally-scoped knowledge queries? here, we design targeted queries to recall stored temporal facts (section 4.2) and further explore the ability of lms to recall implicit temporal facts (section 4.3). results show that pretrained lms can transfer stored temporallyscoped knowledge to new queries with similar semantics even if the query templates are not observed during training. moreover, we show that with prompts like "from st to et", lms can understand the difference and continuity of temporal scopes. these results show that lms can efficiently handle temporal knowledge. contributions: (1) we introduce three challenges and two practical requirements for treating lms as temporal kbs, which expands the lm-as-kb paradigm @xcite to the temporal domain. (2) we offer lama-tk, a new dataset for probing lms for temporally-scoped knowledge. (3) we propose a prompt-based temporal scope modeling method to jointly model temporal scopes and facts for adapting lms to temporallyscoped facts. (4) we conduct experiments to evalu-ate the capacity of lms to store temporal facts and examine the ability of lms to use stored knowledge for temporal queries. (5) we show the negative impact of conflicting information on the storage capacity of lms, which was neglected by previous works. ## the lama-tk probe in this section, we detail the construction of lama-tk foot_0 , our new temporally-scoped knowledge probing dataset, including its data sources and a set of natural language queries for probing temporal knowledge, as well as the evaluation metric we use. ## models following heinzerling and inui (2021), we use roberta @xcite , the bidirectional lm, as the knowledge base. moreover, we adopt several approaches to adapt the original roberta to temporally-scoped knowledge and prepare three different roberta models for examination. ## experiments storage and the use of stored knowledge are two basic functions of kbs. to explore the lm-as-kb paradigm in the temporal domain, we design several experiments to answer the two questions of lms as temporal kbs (section 1). first, we conduct a reciting experiment to evaluate the storage capacity of lms for storing temporal facts and explore the impact of conflicting information (section 4.1). next, we construct targeted queries to recall stored temporal knowledge in terms of temporal boundaries (section 4.2) and implicit temporal knowledge (section 4.3) to explore the ability of lms to use stored knowledge. ## related work recent research has shown that pretrained language models (plms) such as bert @xcite , roberta @xcite , gpt @xcite , and t5 @xcite can learn extensive world knowledge during pretraining and store these relational facts in their parameters. @xcite constructs lama, a set of cloze-style queries (e.g., "marcello abbado was born in @xcite . -> milan"), to recall the factual knowledge contained in pretrained lms such as elmo @xcite and bert @xcite . their results show that a plm contains relational knowledge and can recall stored facts without fine-tuning. @xcite proposes eight cloze-style reasoning tasks to test different types of knowledge in bert and roberta. @xcite conduct experiments on roberta to evaluate its ability to store millions of facts involving millions of entities and its ability to query stored facts. its results provide a proof-of-concept for lm-as-kb. while these works focus on probing lm in the general domain, @xcite constructs the biolama, a biomedical factual knowledge dataset for probing biomedical lms, and further explores the capability of lm to act as a specific-domain kb. moreover, @xcite ; @xcite examine plms on commonsense reasoning tasks, showing that plm contains commonsense knowledge. to improve the performance on knowledge intensive tasks, @xcite uses a transformer encoder to obtain contextualized entity and relation embeddings. @xcite treats relational knowledge as textual sequences and finetunes bert to model these knowledge. to improve the performance of recalling knowledge, @xcite augments plm with retrieved relevant context and improved the performance of cloze-style qa. @xcite proposes mining-based and paraphrasing-based methods to generate high-quality prompts, which significantly improves the performance achieved on lama. within the current paradigm of the use of masked lms as kbs, research has focused more on using generative lms as kbs. as generative lms can generate text sequences of any length, they are not limited by the length of the given knowledge. @xcite fine-tunes the pretrained t5 model to three qa datasets web-questions @xcite , triviaqa @xcite , and naturalquestions @xcite without any access to external knowledge to test how much knowledge contained in the lm. the results show that fine-tuned t5 performs competitively with retrieval-based systems, and indicate that large pretrained lms contain vast world knowledge. @xcite argues that lms can complete the closed-book qa tasks well, mostly due to high test-train overlaps. @xcite designs a knowledge memory task and a question-answering task on datasets with low testtrain overlaps to evaluate the capability of bart @xcite to serve as a kb for closedbook qa. the results show that closed-book qa is still challenging for bart, both in terms of memorizing the knowledge and answering the questions. @xcite proposes a time-aware t5 model which jointly models the text with its timestamp, and constructs a new dataset called templama for probing lms for temporal facts. apart from closed-book qa, @xcite examines cloze tasks for bert to identify the neurons that store specific facts. the results demonstrate the provenance of specific knowledge in the parameters of an lm. @xcite focus on editing stored knowledge without affecting other facts. these works further explore the ability of lms and expand their functions as kbs. ## conclusion temporal knowledge widely exists in real-world kbs. in this work, we extend the lm-as-kb paradigm to the temporal domain and argue that pretrained lms have fairly good capability to serve as temporal knowledge bases in terms of their capacity to store temporal knowledge and their ability to use stored temporal knowledge. however, our analysis also shows that conflicting information poses great challenges to the lm-as-kb paradigm, such as the drop in storage accuracy and the difficulty in memorizing multiple answers.