Are LLM s Good Annotators for Discourse-level Event Relation Extraction? (2024)

Abstract: Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks. However, its effectiveness over discourse-level event relation extraction (ERE) tasks remains unexplored. In this paper, we assess the effectiveness of LLMs in addressing discourse-level ERE tasks characterized by lengthy documents and intricate relations encompassing coreference, temporal, causal, and subevent types. Evaluation is conducted using an commercial model, GPT-3.5, and an open-source model, LLaMA-2. Our study reveals a notable underperformance of LLMs compared to the baseline established through supervised learning. Although Supervised Fine-Tuning (SFT) can improve LLMs performance, it does not scale well compared to the smaller supervised baseline model. Our quantitative and qualitative analysis shows that LLMs have several weaknesses when applied for extracting event relations, including a tendency to fabricate event mentions, and failures to capture transitivity rules among relations, detect long distance relations, or comprehend contexts with dense event mentions.

## introduction event relation extraction (ere) refers to the nlp tasks that identify and classify relationships between events mentioned in a text. the commonly studied event relations include coreference, temporal, causal and subevent relations. ere tasks aim to comprehend the intricate relationships between events and are beneficial for many applications, such as event prediction @xcite , question answering @xcite , and reading comprehension @xcite . ere tasks remain difficult and the empirical performance on these tasks are often rather low. recently, inspired by the recent success of llms, the [ expedition event_0 ] of the thousand ( italian "spedizione dei mille " ) was an event of the italian risorgimento that [ took place event_1 ] in [ 1860 timex_0 ]. a corps of volunteers led by giuseppe garibaldi [ sailed event_2 ] from quarto, … [ ruled event_5 ] by the house of bourbon-two sicilies. the project was an ambitious and risky [ venture event_6] [ aiming event_7 ] to [ conquer event_8 ], with a thousand men, a kingdom with a larger regular army and a more powerful navy. coreference: event_6 coreference event_0; event_8 coreference event_4; event_16 coreference event_15; … temporal: event_0 before event_2; event_2 before event_5; timex_0 after event_1; … event_5 before event_0; causal: event_3 precondition event_4; event_3 cause event_4; … subevent: event_0 subevent event_2; event_0 subevent event_10; shift; ... @xcite evaluates chatgpt foot_0 on one ere task, temporal relation extraction, using several benchmark datasets, including tddiscourse @xcite that addresses temporal relation extraction at the discourse level. however, @xcite has to truncate the documents and constrain the size of discourse as inputs longer than eight sentences cause chatgpt to generate unformatted answers. meanwhile, @xcite only evaluates chatgpt on its causal reasoning ability. unlike these works, we evaluate llms on performing multiple ere tasks at the document level that tend to feature dense and long distance event relations, as shown in figure 1 . specifically, we experiment with two llms, the commercial model and the open-source model llama-2. furthermore, humans usually benefit from comprehending the meanings of event relations through examples @xcite when conducting the ere tasks, but the recent studies @xcite often evaluate llms under the zero-shot setting and have a chance to limit the models in fully showcasing their true capabilities in event relation extraction. to fill the gap, we run experiments in the one-shot or few-shot settings, we also run supervised fine-tuning (sft) experiments aiming to further enhance llms on performing ere tasks. we systematically evaluate the effectiveness of llms on extracting four common types of event relations (coreference, temporal, causal, and subevent) using the maven-ere dataset @xcite by experimenting with multiple prompts. the design of our prompts were informed by the prior work @xcite to ensure the most effective prompts applied. comprehensive analysis, both quantitative and qualitative, lead to the following findings: furthermore, llms display inconsistencies in adhering to the provided prompts. both suggest a potential lack of reasoning abilities and inaccurate understanding on the assigned task. ## related works event relation extraction event relation extraction (ere) has been one of the fundamen-tal challenges for natural language processing @xcite @xcite . as understanding relations between events is crucial for understanding human languages @xcite and beneficial for various applications @xcite @xcite , many approaches have been developed for performing ere tasks @xcite @xcite , and many high performing approaches are based on supervised learning @xcite @xcite . llms for extraction tasks llms have been applied to several common information extraction tasks including event extraction, relation extraction and named entity recognition @xcite @xcite @xcite . however, to the best of our knowledge prompt engineering many recent works have studied how to make llms perform better through applying various prompting techniques, including role-prompting @xcite , re-sampling @xcite , oneshot or few-shot prompting @xcite , and question decomposition @xcite . other novel and advanced techniques include chain of thought prompting @xcite , least-to-most prompting @xcite , learning from multiple prompts @xcite , and retrieval augmentation @xcite . we refer to these techniques as guidelines when designing prompts in this work. ## results llms performance we report the performance of gpt-3.5 and llama-2 on extracting coreference, temporal, subevent, and causal relations in for coreference relations, we report results using four metrics following previous work @xcite . regarding temporal, causal, and subevent relations, gpt-3.5's performance increases as the number of example documents increases for n-shot prompt with the 10-shot prompt yielding the best performance. while for llama-2, the whole doc prompt yields the best performance. regarding individual types of relations, both models have the best performance on temporal relations and the worst performance on subevent relations. we believe such a performance gap is primarily because temporal relations are much denser than subevent relations in maven-ere . the last column of figure 3 shows the result comparisons between llama-2 and the smaller baseline model. first, sft certainly improves the performance of llama-2. however, it still underperforms the smaller baseline method for all the relation types as the number of used training documents increases. llama-2 typically requires twice more training data to reach the same overall performance as the smaller baseline model. llama-2 only outperforms the smaller baseline method when the available training data is very limited, as llama-2 benefits from its zero-shot capability emergent from large-scale pre-training. it also deserves mentioning that fine-tuning llms is much more expensive than fine-tuning the smaller baseline language model roberta. for example, fine-tuning using 200 training documents for 3 epochs requires approximately 72 hours for llama-2 7b, while only about one hour is needed for the roberta-base baseline model. ## conclusion in this study, we systematically evaluated the effectiveness of llms in performing discourse-level ere tasks featuring lengthy documents and intricate relations. our experiments using multiple prompt patterns uncover a noteworthy underperformance of llms when compared to the baseline established through supervised learning. even with supervised fine-tuning, llms like llama-2 still underperform the much smaller supervised baseline model when trained on the same amount of data. furthermore, our quantitative and qualitative analyses revealed that llms face challenges in obeying transitivity rules, capturing inter-sentence relations or relations with a long distance, as well as comprehending context with dense event mentions. for future work, we will further investigate these challenges and develop methods for enabling llms to better address some of these issues in event relation extraction.