Timeline-based Sentence Decomposition with In Context Learning for Temporal Fact Extraction (2024)

Abstract: Facts extraction is pivotal for constructing knowledge graphs. Recently, the increasing demand for temporal facts in downstream tasks has led to the emergence of the task of temporal fact extraction. In this paper, we specifically address the extraction of temporal facts from natural language text. Previous studies fail to handle the challenge of establishing time-to-fact correspondences in complex sentences. To overcome this hurdle, we propose a timeline-based sentence decomposition strategy using large language models (LLMs) with in-context learning, ensuring a fine-grained understanding of the timeline associated with various facts. In addition, we evaluate the performance of LLMs for direct temporal fact extraction and get unsatisfactory results. To this end, we introduce TSDRE, a method that incorporates the decomposition capabilities of LLMs into the traditional fine-tuning of smaller pre-trained language models (PLMs). To support the evaluation, we construct ComplexTRED, a complex temporal fact extraction dataset. Our experiments show that TSDRE achieves state-of-the-art results on both HyperRED-Temporal and ComplexTRED datasets.

## introduction acquiring knowledge has long been a fundamental challenge in the field of artificial intelligence. typically, the acquired knowledge is stored in knowledge graphs (kgs) as triples, comprising a head entity, a relation, and a tail entity. recently, there has been a significant surge in the need @xcite @xcite @xcite @xcite for acquiring temporal facts. for instance, users may seek information from the kgs, such as "when did michael jordan win the nba finals mvp?". however, traditional triples like (nba finals mvp, winner, michael jordan) cannot meet such needs due to the lack of time dimension. this can be solved by adding time information like @xcite to the triple. a temporal fact can be formalized as a quintuple (e head , r, e tail , q, t), where the time qualifier q and time value t indicate the time dimension information of the triple. although many works @xcite @xcite have explored the task of temporal fact extraction, the challenge of establishing correspondence between time and triples remains unresolved. figure 1 shows a difficult example of temporal fact extraction. it can be seen that the difficulty of this example mainly comes from the use of "the other" to imply that the other two players also won three awards in one year. besides, natural language expressions are typically succinct, with "michael jordan in 1996 and 1998" alone expressing six temporal facts. existing methods cannot handle this situation without explicitly stating what event occurred at a certain point in time, leading to suboptimal performance in addressing complex narratives with interwoven timelines. to address this gap, we propose a timelinebased sentence decomposition strategy. our strategy involves breaking down sentences according to their timelines to capture the temporal dimensions of facts, ensuring a fine-grained understanding of the timeline associated with various facts. in the past, sentence decomposition often required a large amount of training corpus. but now, the incontext learning capability of large language models (llms) empowers us to perform timeline-based sentence decomposition without training corpus. to the best of our knowledge, we are the first to investigate the application of llms for temporal fact extraction tasks. we conduct an evaluation to assess the performance of employing llms directly for temporal fact extraction. however, it is unsatisfactory that the performance of llms does not surpass the traditional approach of fine-tuning smaller plms. to enhance the performance of temporal fact extraction methods, we come up with the idea of combining the surprising sentence decomposition capability of llms with the traditional way of fine-tuning small pre-trained language models. experiments show that our method based on the combining idea achieves sota results. in addition to research on extraction methods, there are currently few benchmarks @xcite for temporal fact extraction, and they do not pay much attention to sentences involving complex time related narratives (we call them complex sentences for short). to this end, we construct a temporal fact extraction dataset composed of complex sentences for evaluation. in summary, our main contributions in this paper are outlined as follows: ## method in this section, we begin by presenting two direct approaches to leverage llms for temporal fact extraction. following that, we introduce our extraction methods, encompassing a timeline-based sentence decomposition strategy and the fine-tuning of generative models using decomposition results for training. time recognition 2000 1970 1996 1998 input prompt: extract the quintuples ['head entity', 'relation', 'tail entity', 'qualifier', 'time point'] from the input text. the output format is a list of quintuples. timeline gives the results of the events in the text sorted according to the timeline to assist in your extraction. text: shaquille o'neal is one of only three… timeline: input prompt: extract the quintuples ['head entity', 'relation', 'tail entity', 'qualifier', 'time point'] from the input text. the output format is a list of quintuples. text: shaquille o'neal is one of only three players … flan-t5 output: ['nba mvp'/'all-star game mvp'/'finals mvp', 'winner', 'shaquille o'neal', 'point in time', '2000'] ## complextred: a complex temporal fact extraction dataset there are very few temporal fact extraction datasets available to date. pravda @xcite lacks annotation labels and therefore cannot be used for supervised learning. the wiki-people dataset @xcite is not open source. hyperred (chia et al., 2022) is a hyper-relational fact extraction dataset, 48% of which are temporal facts. however, many samples in hyperred are overly simplistic and insufficient to depict the potential difficulties in complex practical scenarios. specifically, the majority of sentences in hyperred consist solely of a single temporal expression, and their respective extraction results comprise a singular temporal fact. we need a complex temporal fact extraction dataset to evaluate our method and train existing models on their ability to extract temporal facts from complex temporal sentences. below we will introduce data collection and dataset statistics. ## conclusion in this paper, we explore the application of large language models (llms) in the extraction of temporal facts. our attempts indicate that directly employing llms for temporal fact extraction falls short of achieving satisfactory results. to tackle this issue, we introduce a timeline-based sentence decomposition (tsd) method. building upon this, we propose tsdre, which employs a relatively smaller plm as its foundation, combined with llm-driven tsd to achieve the extraction. experiments demonstrate that tsdre achieves sota results on two datasets and incorporating tsd into the training process can enhance the performance of llms on temporal fact extraction tasks. in the future, an interesting topic would be to explore the extraction of temporal facts from text that necessitate inferring the occurrence time based on existing temporal references, such as "three days later", which has not yet received widespread attention.