Exploring the Potential of C hat GPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations (2024)

Abstract: This paper aims to quantitatively evaluate the performance of ChatGPT, an interactive large language model, on inter-sentential relations such as temporal relations, causal relations, and discourse relations. Given ChatGPT’s promising performance across various tasks, we proceed to carry out thorough evaluations on the whole test sets of 11 datasets, including temporal and causal relations, PDTB2.0-based, and dialogue-based discourse relations. To ensure the reliability of our findings, we employ three tailored prompt templates for each task, including the zero-shot prompt template, zero-shot prompt engineering (PE) template, and in-context learning (ICL) prompt template, to establish the initial baseline scores for all popular sentence-pair relation classification tasks for the first time. Through our study, we discover that ChatGPT exhibits exceptional proficiency in detecting and reasoning about causal relations, albeit it may not possess the same level of expertise in identifying the temporal order between two events. While it is capable of identifying the majority of discourse relations with existing explicit discourse connectives, the implicit discourse relation remains a formidable challenge. Concurrently, ChatGPT demonstrates subpar performance in the dialogue discourse parsing task that requires structural understanding in a dialogue before being aware of the discourse relation.

## introduction with the proliferation of computational resources and the availability of extensive text corpora, the expeditious advancement of large language models (e.g., @xcite and @xcite ) have prominently showcased their emergence ability resulting from the scaling up model size. techniques such as instruction tuning @xcite and reinforcement learning from human feedback @xcite have further fortified llm with sophisticated language understanding and logical reasoning proficiencies. therefore, these large language models (llms) demonstrate remarkable few-shot, even zero-shot learning abilities in performing various tasks. recent studies have extensively and comprehensively evaluated chatgpt's performance on numerous language understanding and reasoning tasks, revealing that its superior performance in zero-shot scenarios when compared to other models @xcite @xcite . besides, chatgpt has also shown impressive powers in data annotations and has proven to be more cost-efficient than crowdworkers for several annotation tasks @xcite . whilst the success of chatgpt has been witnessed, certain obstacles persist unaddressed. previous research has discussed the associated ethical implications and privacy concerns @xcite @xcite . moreover, chatgpt's shortcomings include but are not limited to the lack of planning @xcite , the inability to perform complex mathematical reasoning @xcite , and fact validation @xcite @xcite . consequently, it is still under discussion whether large language models possess the capacity to comprehend text beyond surface forms as humans. to comprehend the natural language text at a deeper level, it is crucial for an llm to capture and understand the higher-level inter-sentential relations from the text, which involves mastering more complex and abstract relations beyond surface forms. these inter-sentential relations, such as temporal, causal, and discourse relations between two sentences, are widely used to form knowledge that has been proven to benefit many downstream tasks @xcite @xcite . in this study, we quan-titatively evaluate the performance of chatgpt in tasks that require an understanding of sentencelevel relations, including temporal relation (section 4), causal relation (section 5), and discourse relation (section 6). under three standard prompt settings 2 , we conduct extensive evaluations on the whole test sets of 11 datasets regarding these relations. 3 furthermore, we conducted an in-depth study on the various intra-relations of each intersentential relation (e.g., before and after relation in temporal relations) and assessed the performance of the chatgpt on these specific intra-relations. the detailed relation-wise performance is shown in figure 1 . the primary insights drawn from the analysis of quantitative assessments are as follows 4 : it also outperforms fine-tuned roberta on two out of three benchmarks. we aspire to contribute to the research community through our evaluations and discoveries. by sharing the result, we intend to offer valuable insights to others in the relevant fields. temporal relation temporal relation extraction aims to detect the temporal relation between two event triggers in the given document @xcite . it is crucial for many downstream nlp tasks since reasoning over temporal relations plays an essential role in identifying the timing of events, estimating the duration of activities, and summarizing the chronological order of a series of occurrences @xcite . there exists a recent work that evaluates chatgpt's ability on zero-shot temporal relation extraction @xcite . however, their manually designed prompts acquire unsatisfiable performance, and the capability of chatgpt equipped with in-context learning has not been explored. therefore, this work also includes the temporal relation tasks, and our results can complement and validate each other with @xcite . causal relation causal reasoning involves the identification of causality, which refers to the connection between a cause and its corresponding effect @xcite . nlp models that can reason causally have the potential to improve their ability to understand language, as well as to solve complex problems in various fields, such as physical reasoning @xcite , event extraction @xcite , question-answering @xcite , and text classification @xcite . although @xcite has analyzed chatgpt's performance in a medical causality benchmark, no prior research has conducted a comprehensive study on the ability of large language models to reason upon causal relations. discourse relation discourse relation recognition is a vital task in discourse parsing, identifying the relations between two arguments (i.e., sentences or clauses) in the discourse structure. it is essential for textual coherence and is regarded as a critical step in constructing a knowledge graph @xcite and various downstream tasks involving more context, such as text generation @xcite ), text categorization @xcite , and question answering @xcite . explicit discourse relation recognition (edrr) has already shown that utilizing explicit connective information can effectively determine the types of discourse relations @xcite . in contrast, implicit discourse relation recognition (idrr) remains challenging because of the absence of connectives. however, previous works have not systemically evaluated the ability of chatgpt on these two discourse relation recognition tasks. therefore, in this work, we assess the performance of this large language model (i.e., chatgpt) on the pdtb-style discourse relation recognition task @xcite , dialogue discourse parsing @xcite , and downstream applications on discourse understanding. ## experimental setting we employ three customized prompt templates for each task: zero-shot setting, zero-shot with prompt engineering (pe), and the in-context learning (icl) setting. the devised prompt template will serve as comprehensive and reliable baselines to exclude the variance of the prompt engineering and offer fair comparison baselines for all prevalent sentence-pair relation classification tasks. the specific template details are presented in corresponding sections and appendix c. ## temporal relation temporal relation extraction aims to determine the temporal order between two events in a text @xcite , which could be formulated as a multi-label classification problem. in this section, we evaluate the temporal reasoning ability of chatgpt on three commonly used benchmarks: tb-dense @xcite , matres @xcite , and tddman @xcite (details in appendix a). to ensure compatibility with previous research, we employ the same data split and assess chatgpt's performance on the entire test set. ## causal relation causal reasoning is the process of understanding and explaining the cause-and-effect relationships between events @xcite . it involves identifying the factors that contribute to a particular result and understanding how changes in those factors can lead to different outcomes @xcite . in this paper, we assess the causal reasoning ability of llms by benchmarking their results on three existing causal reasoning datasets (copa @xcite , e-care @xcite , and headlinecause @xcite , details in appendix a) and quantitatively analyzing the results. our findings demonstrate that the llm exhibits a robust ability to detect and reason about causal relationships, particularly those pertaining to cause and effect, without requiring advanced prompting techniques such as in-context learning. ## discourse relation in this section, we evaluate chatgpt on discourse relation recognition tasks, including pdtb-style discourse relation recognition, multi-genre crowd-sourced discourse relation recognition, dialogue discourse parsing, and applications on discourse understanding. apart from these datasets and tasks, we conduct the assessments of chat-gpt's performance on two downstream tasks which are shown in appendix b. ## conclusion and future work in conclusion, this study thoroughly examines chat-gpt's ability to handle pair-wise temporal relations, causal relations, and discourse relations by assessing its performance on the complete test sets of over 11 datasets. the result exhibits that even though chatgpt obtains impressive zero-shot performance across other various tasks, there is still a gap for chatgpt to achieve excellent performance on temporal and discourse relations. though there may be numerous other capabilities of chatgpt that go unnoticed in this paper, future work should nonetheless investigate the capability of chatgpt on more tasks (e.g., analogy relation between two sentences @xcite ).