Improving Large Language Models in Event Relation Logical Prediction (2024)

Abstract: Event relations are crucial for narrative understanding and reasoning. Governed by nuanced logic, event relation extraction (ERE) is a challenging task that demands thorough semantic understanding and rigorous logical reasoning. In this paper, we conduct an in-depth investigation to systematically explore the capability of LLMs in understanding and applying event relation logic. More in detail, we first investigate the deficiencies of LLMs in logical reasoning across different tasks. Our study reveals that LLMs are not logically consistent reasoners, which results in their suboptimal performance on tasks that need rigorous reasoning. To address this, we explore three different approaches to endow LLMs with event relation logic, and thus enable them to generate more coherent answers across various scenarios. Based on our approach, we also contribute a synthesized dataset (LLM-ERL) involving high-order reasoning for evaluation and fine-tuning. Extensive quantitative and qualitative analyses on different tasks also validate the effectiveness of our approach and provide insights for solving practical tasks with LLMs in future work. Codes are available at https://github.com/chenmeiqii/Teach-LLM-LR.

## introduction understanding the relationships between events is fundamental to effective communication and reasoning, a challenge central to the field of event relation extraction (ere). ere tasks, which involve identifying coreference, temporal, causal, and subevent relationships, demand not only semantic comprehension but also rigorous logical reasoning. despite recent advances in large language models (llms) such as chatgpt @xcite identify the relations between events fire and collapsed … text: a large fire broke out at the waitrose supermarket in wellington's high street, half of the roof at the entrance of the store collapsed during the blaze. coreference relation: no_coreference temporal relation: simultaneous causal relation: cause subevent relation: no_subevent. ## unveiling llms in logical reasoning considering the rigorous logical reasoning required by ere tasks, in this section, we conduct a pilot study to investigate how current llms exhibit reasoning tasks and how logic benefits llms. ## teaching llms to predict event relation logic from the above analysis, the main reason for the failure of llms stems from their lack of logical reasoning abilities. in this section, we expect to explore how to augment llms with the capability to comprehend and apply event relation logic. specifically, we first introduce the instruction-following technique used in section 4.1 and then propose three different approaches to instruct llms to generate answers with better logical consistency, including generative-based, retrieval-based, and finetuning-based approaches (section 4.2 to 4.4). we illustrate these three approaches in figure 4 . ## conclusion in this paper, we conduct a detailed investigation on how to enhance llms with event relation logic. specifically, we first investigate the existing issues of current llms in event relation logical prediction. then, we study multiple strategies to obtain and utilize logic for llms, including generative-based, retrieval-based, and finetuning-based approaches. based on our approaches, we also contribute a synthesized dataset (llm-erl) involving multi-hop reasoning for evaluation and fine-tuning. we show that llms are not logically consistent reasoners, but their performance could be improved if we explicitly teach them the logical constraints. comprehensive quantitative and qualitative analyses have been conducted to further provide insights.