Once Upon a {\it Time} in {\it Graph} : Relative-Time Pretraining for Complex Temporal Reasoning (2023)

Abstract: Our physical world is constantly evolving over time, rendering challenges for pre-trained language models to understand and reason over the temporal contexts of texts. Existing work focuses on strengthening the direct association between a piece of text and its time-stamp. However, the knowledge-time association is usually insufficient for the downstream tasks that require reasoning over temporal dependencies between knowledge. In this work, we make use of the underlying nature of time, all temporally-scoped sentences are strung together through a one-dimensional time axis, and suggest creating a graph structure based on the relative placements of events along the time axis. Inspired by the graph view, we propose RemeMo ( \underline{Re}lative Ti\underline{me} \underline{Mo}deling ), which explicitly connects all temporally-scoped facts by modeling the time relations between any two sentences. Experimental results show that RemeMo outperforms the baseline T5 on multiple temporal question answering datasets under various settings. Further analysis suggests that RemeMo is especially good at modeling long-range complex temporal dependencies.

## introduction knowledge plays a crucial role in downstream nlp applications. pre-trained language models (plms) generate accurate responses to various user queries by reasoning over their internal knowledge and the knowledge contained in texts. real-world knowledge, however, frequently expires and updates. taking an example in to enhance the temporal understanding capability of plms, recent studies have focused on strengthening the one-to-one associations between an event and its corresponding time tokens or time stamps @xcite @xcite . by modeling such superficial temporal dependencies, these methods gain improvements on various time-sensitive tasks. these methods, however, assume all temporallyscoped facts as independent of each other, ignoring the complex temporal dependencies among temporally-scoped facts. this may pose limitations because time-related queries often require reasoning over multiple temporally-scoped facts. for example, in the most well-known concept of time is the linear concept, which views time as a series of moments that unfold sequentially from the past, through the present, and into the future. as shown in figure 1a , temporally-scoped events are strung together through the one-dimensional time axis. by inspecting the relative positions between any two events, we can build a fully-connected digraph, where events act as graph nodes and temporal relations among them serve as directed graph edges, as shown in figure 1b . complex temporal dependencies can thus be systematically modeled from the perspective of graph learning. for example, as shown in figure 1b , fact-1 (as the start node) is associated with fact-2 (as the destination node) either directly through the earlier edge or indirectly by passing through two edges via fact-3. inspired by this graph view, we present relative-time modeling (rememo): a novel time-aware pre-training framework that exploits relative positions on the time-axis to model complex temporal dependencies. specifically, we extract time-span tags from sentences and then pre-train the plm with two joint objectives: (i) the language modeling (lm) objective; (ii) a novel time relation classification (trc) objective, which classifies the relation between any two sentences, i.e., sentence-a is earlier, later or contemporary foot_0 compared with sentence-b. from the graph learning perspec-tive, the lm objective can be seen as node feature learning, while the trc objective operates as edge classification. rememo is pre-trained on regular corpora and temporally-scoped sentences extracted from wikipedia articles. it is then evaluated on seven downstream temporal open-book question answering datasets. rememo consistently gives better results compared with the baseline t5 model under multiple settings. overall, our contributions are summarized as follows: extensive experiments show that rememo gains significant improvements over t5 baselines under various settings. ## related work temporal question answering temporal question answering has attracted much research attention in recent years. an early line of work focused on temporal question answering over knowledge graphs, for which several datasets were built, such as tempquestions @xcite , tequila @xcite , timequestions @xcite and cronkgqa @xcite . under this temporal kgqa setting, the model is required to do complex temporal reasoning over temporal knowledge graphs and to select an entity node from a given node list as the answer. more recently, another line of work has focused on temporal question answering either under the closed-book setting to evaluate models' internal memorization of temporal facts or under the open-book setting to evaluate models' temporal understanding and reasoning capability over unstructured texts. some example datasets include templama @xcite , situat-edqa @xcite , timeqa @xcite , archivalqa @xcite . in 2010, james famously announced his decision to join the miami heat. golden state warriors defeated the cleveland cavaliers in six games in the 2015 nba finals. james returned to the cleveland cavaliers in 2014 and led the team to their first ever nba championship in 2016. ## rememo motivated by the graph view shown in figure 1b , we present rememo to model the temporal dependencies among temporally-scoped sentences 2 . specifically, given a document containing multiple sentences, we first apply a time-identification & time-normalization pipeline to assign a time-span tag for each sentence ( §3.1). time-relation labels are then created over any pair of temporally-scoped sentences according to the chronological order of the events/facts in the sentences and will be utilized for later time-aware pre-training. pre-trained on both the language modeling (lm) objective and the time relation classification (trc) objective ( §3.2), rememo is capable of capturing complex temporal dependencies beyond simple time-token co-occurrences. finally, we discuss variants of re-memo by controlling the graph density of the trc objective ( §3.3). ## experiments we present the empirical study in this section. experimental settings are in § 4.1 and § 4.2. main results are in § 4.3. we further investigate the challenges of modeling complex temporal dependencies in § 4.4. ## conclusions in this paper, we devised a graph view of temporally-scoped events based on the fact that these events are strung together through the onedimensional time axis. inspired by this graph view, we proposed the trc pre-training objective that exploits relative time as directed graph edges for complex temporal reasoning. we pre-trained re-memo jointly with the lm objective and the trc objective and evaluated it on seven downstream temporal qa datasets. experimental results show that rememo outperforms the baseline t5 under various settings. the performance improvements are remarkably large on datasets that require reasoning over complex "event-event" temporal relations. further analysis suggests that (i) rememo is especially good at modeling long-range complex temporal dependencies; (ii) the key to the success of rememo is the graph-inspired modeling rather than the larger density of the graph.