MAVEN - FACT : A Large-scale Event Factuality Detection Dataset (2024)

Abstract: Event Factuality Detection (EFD) task determines the factuality of textual events, i.e., classifying whether an event is a fact, possibility, or impossibility, which is essential for faithfully understanding and utilizing event knowledge. However, due to the lack of high-quality large-scale data, event factuality detection is under-explored in event understanding research, which limits the development of EFD community. To address these issues and provide faithful event understanding, we introduce MAVEN-FACT, a large-scale and high-quality EFD dataset based on the MAVEN dataset. MAVEN-FACT includes factuality annotations of 112,276 events, making it the largest EFD dataset. Extensive experiments demonstrate that MAVEN-FACT is challenging for both conventional fine-tuned models and large language models (LLMs). Thanks to the comprehensive annotations of event arguments and relations in MAVEN, MAVEN-FACT also supports some further analyses and we find that adopting event arguments and relations helps in event factuality detection for fine-tuned models but does not benefit LLMs. Furthermore, we preliminarily study an application case of event factuality detection and find it helps in mitigating event-related hallucination in LLMs. We will release our dataset and codes to facilitate further research on event factuality detection.

## introduction event factuality detection (efd) aims to determine the factuality of textual events, i.e., classifying whether an event is a fact, possibility, or impossibility @xcite @xcite @xcite . as shown in figure 1 , the event "play" is a fact while the event "celebrate" is just a possibility considering the word "might". * equal contribution. ## dataset construction this section introduces the definition of event factuality detection ( § 2.1), the llm-then-human annotation approach ( § § 2.2 to 2.4), and data analysis of maven-fact ( § 2.5). ## mitigating event-related hallucinations in addition to benchmarking efd models, we also want to explore potential application scenarios of maven-fact. here, we preliminarily explore using event factuality to mitigate event-related hallucinations in llms, as non-factuality is a primary source of hallucination @xcite . hallucination refers to the phenomenon where the outputs of models do not align with the input, typically involving non-factual information in the document: the 2014 bukidnon bus bombing occurred on december 9, 2014. ... extortion is viewed as a motive for the attacks due to claims that the bus company has faced threats for refusing to pay protection money to the militants. the militant group denies any involvement claiming they would not gain any benefit from conducting such attacks and claims the accusations against them as fabrication. question: did the bus bombing occur because the bus company refused to pay protection money to the militants? answer: no experimental setup we begin with constructing a knowledge-intensive question-answering (qa) dataset based on maven-fact, which is a scenario susceptible to hallucinations @xcite . as we aim to analyze hallucination in llms, we deliberately craft questions that are prone to induce hallucination. specifically, we first sample 800 documents from the maven-fact test set. for each document, we select the event which is non-factual and have the most relation connections with other events in the document. 2 we then utilize gpt-4 to generate yes-or-no questions that require complex reasoning along with the answers for each event based on its mentioned document. then three experts manually review all the questions and answers, and eliminate questions without answers or not requiring reasoning, and correct erroneous answers, resulting in 450 validated instances. an example of the data is shown in ## conclusion this paper introduces maven-fact, the largest and high-quality event factuality detection dataset. maven-fact comprehensively includes supporting evidence for factuality and event annotations from maven. experimental results demonstrate that maven-fact poses a significant challenge to efd models and llms. we also find that using event factuality can help in mitigating event-related hallucinations in llms. we hope that maven-fact will facilitate research on the development and application of event factuality detection.