Zero-shot Temporal Relation Extraction with C hat GPT (2023)

Abstract: The goal of temporal relation extraction is to infer the temporal relation between two events in the document. Supervised models are dominant in this task. In this work, we investigate ChatGPT’s ability on zero-shot temporal relation extraction. We designed three different prompt techniques to break down the task and evaluate ChatGPT. Our experiments show that ChatGPT’s performance has a large gap with that of supervised methods and can heavily rely on the design of prompts. We further demonstrate that ChatGPT can infer more small relation classes correctly than supervised methods. The current shortcomings of ChatGPT on temporal relation extraction are also discussed in this paper. We found that ChatGPT cannot keep consistency during temporal inference and it fails in actively long-dependency temporal inference.

## introduction the temporal relation extraction task aims to extract the temporal relation between either two event triggers in the given document @xcite . in this way, a timeline of events in the document can be constructed. it is a crucial task for many downstream nlp tasks, such as natural language understanding @xcite , storyline construction @xcite , and temporal question answering @xcite , etc. conventionally, recent temporal relation extraction (re) models are fine-tuned based on pre-trained language models (plms), such as bert and roberta @xcite . on top of the plms, complex neural networks are applied to classify the temporal relations, such as self-attention @xcite , graph convolutional networks (gcns) @xcite , and policy network @xcite . most well-performed temporal relation extractors are supervised models, that is, they heavily rely on annotated training documents first before extracting temporal relations on the testing set. however, annotating the temporal relations in training documents requires much domain experts' efforts @xcite , which is a high cost. different from supervised learning methods, zero-shot learning (zsl) @xcite aims to train the model that can be generalized to unseen data without annotated training data and has attracted much attention in recent years. most recently, large language models (llms) @xcite such as chat-gpt foot_0 have exhibited remarkable ability in zeroshot learning on various natural language processing (nlp) and medical tasks @xcite , such as information extraction @xcite , machine translation @xcite , summarization evaluation @xcite , and mental health detection @xcite . however, the performance of llms in detecting temporal relations between events are not explored yet. therefore, it is an urgent and spontaneous question if the llm can perform zero-shot temporal relation extraction tasks well given a proper prompt approach, and if llm can be the new paradigm for temporal re. in this paper, we explore the performance of chatgpt on the zero-shot temporal relation extraction, and propose three different prompt strategies to interact with chatgpt. more specifically, we start with the simple zero-shot prompt that directly requires chatgpt to infer the temporal relation given the document. then, we design the event ranking prompt, where chatgpt is asked to infer the events shown in the given document instead of inferring temporal relations. finally, we propose the chain-of-thought (cot) prompt @xcite to break down the task into two-stage, which guides chatgpt to make temporal relation reasoning step by step. based on our experimental results and analysis, we have the following findings: ## chatgpt for zero-shot temporal relation extraction given the input document, the temporal relation extraction aims to identify the temporal relation between any two event triggers in the document, which is modeled as the multi-classification problem. we propose three different prompts methods to evaluate chatgpt's performance on temporal relation extraction as shown in figure 1 . ## conclusion in this work, we comprehensively test chatgpt's zero-shot ability on temporal relation extraction. we designed three different prompts to evaluate chatgpt's performance. our experiments demonstrate that with proper prompting, chatgpt's performance on zero-shot temporal relation extraction can be significantly improved, highlighting the importance of prompt engineering to better trigger chatgpt's ability in future work. however, compared to supervised methods, chatgpt is still far behind. we further discuss our findings from experimental results, including its better performance in small classes than supervised methods and its drawbacks such as failures in long-distance temporal dependency inference and inconsistent temporal relation inference. our work only takes the initial step of exploring the llms for zero-shot temporal relation extraction. to fill the gap between the performance of llms in the zero-shot setting and that of advanced supervised methods, we believe more efforts should be explored in the future.