Fusing Temporal Graphs into Transformers for Time-Sensitive Question Answering (2023)

Abstract: Answering time-sensitive questions from long documents requires temporal reasoning over the times in questions and documents. An important open question is whether large language models can perform such reasoning solely using a provided text document, or whether they can benefit from additional temporal information extracted using other systems. We address this research question by applying existing temporal information extraction systems to construct temporal graphs of events, times, and temporal relations in questions and documents. We then investigate different approaches for fusing these graphs into Transformer models. Experimental results show that our proposed approach for fusing temporal graphs into input text substantially enhances the temporal reasoning capabilities of Transformer models with or without fine-tuning. Additionally, our proposed method outperforms various graph convolution-based approaches and establishes a new state-of-the-art performance on SituatedQA and three splits of TimeQA.

## introduction long-document time-sensitive question answering @xcite requires temporal reasoning over the events and times in a question and an accompanying long context document. answering such questions is a challenging task in natural language processing (nlp) as models must comprehend and interpret the temporal scope of the question as well as the associated temporal information dispersed throughout the long document. for example, consider the time-sensitive questions about george washington's position provided in figure 1 . the relevant events and temporal information regarding george washington's position are scattered across many different sentences in the context document. since there is no one single text segment containing the answer, the model must integrate and reason over events and times ## method our approach applies temporal information extraction systems to construct temporal graphs and then fuses the graphs into pre-trained transformer models. we consider two fusion methods: 1. explicit edge representation fusion (err): a simple but novel approach to fuse the graphs into the input text. 2. graph neural network fusion: a gnn is used to fuse the graphs into the token embeddings or the last hidden layer representations (i.e., contextualized embeddings) of the transformer model. the overall approach is illustrated in figure 2 . ## analysis in this section, we analyze our longt5 err model on the timeqa development set. ## conclusion in this paper, we compared different methods for fusing temporal graphs into transformer models for time-sensitive question answering. we found that our err method, which fuses the temporal graph into the transformer model's input, outperforms gnn-based fusion as well as attention-based fusion models. we also showed that, unlike prior work on temporal graph fusion, our approach is compatible with in-context learning and yields improvements when applied to large language models such as chatgpt. our work establishes a promising research direction on fusing structured graphs with the inputs of transformer models. in future work, we intend to use better-performing information extraction systems to construct our temporal graphs, enhance our approach by adding entities and entity type information to the graphs, and extend our method to spatial reasoning.