D oc EE : A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction (2022)

Abstract: Event extraction aims to identify an event and then extract the arguments participating in the event. Despite the great success in sentence-level event extraction, events are more naturally presented in the form of documents, with event arguments scattered in multiple sentences. However, a major barrier to promote document-level event extraction has been the lack of large-scale and practical training and evaluation datasets. In this paper, we present DocEE, a new document-level event extraction dataset including 27,000+ events, 180,000+ arguments. We highlight three features: large-scale manual annotations, fine-grained argument types and application-oriented settings. Experiments show that there is still a big gap between state-of-the-art models and human beings (41% Vs 85% in F1 score), indicating that DocEE is an open issue. DocEE is now available at https://github.com/tongmeihan1995/DocEE.git .

## introduction event extraction (ee) aims to detect events from text, including event classification and event argument extraction. ee is one of the fundamental tasks in text mining @xcite and has many applications. for instance, it can monitor political or military crises to generate real-time notifications and alerts @xcite , and dig the links and connections (e.g., who met whom and when) between dignitaries for portrait analysis @xcite . most existing datasets (e.g., ace2005 1 and kbp2017 2 ) focus on sentence-level event extraction, while events are usually described at the document level, and event arguments are typically scattered across different sentences @xcite . figure 1 shows an air crash event. to extract argument date, we need to read sentence @xcite , while to extract argument cause of the accident, we need to integrate information in sentences @xcite . clearly, this requires reasoning over multiple sentences and modeling long-distance dependency, intuitively beyond the reach of sentence-level ee. therefore, it is necessary to move ee forward from sentence-level to document-level. only a few datasets are curated for documentlevel ee. muc-4 @xcite provides 1,700 news articles annotated with 4 event types and 5 argument types. the 5 arguments are shared among different event types without further refinement. wikievents @xcite consists of only 246 documents with very few (22% of total) cross-sentences argument annotations. rams @xcite limits the scope of the arguments in a 5-sentence window around its event trigger, which is not in line with the actual application, and the number of the argument types in rams is only 65, which is quite limited. doc2edag, tdjee and git @xcite @xcite contain only 5 event types and 35 argument types in financial domain. in summary, existing datasets for documentlevel ee fail in the following aspects: small scale of data, limited coverage of domain and insufficient refinement of argument types. therefore, it is urgent to develop a manually labeled, large-scale dataset to accelerate the research in document-level ee. in the paper, we present docee, a large-scale human-annotated document-level ee dataset. figure 1 illustrates an example of docee. docee focus on the extraction of the main event, that is oneevent-per-document. we regard news headlines as the main event trigger and focus on main event arguments extraction throughout the article. we highlight the following three contributions of docee to this field: 1) large-scale manual annotations. do- date location casualties and losses survivors investigator of the accident cause of the accident passengers crew aircraft agency flight no. service years taking-off place scheduled landing place alternate landing place ## constructing docee our main goal is to collect a large-scale dataset to promote the development of event extraction from sentence-level to document-level. in the following sections, we will first introduce how to construct the event schema, and then how to collect candidate data and how to label them through crowdsourcing. ## data analysis of docee event type statistic figure 4 shows the distribution of the top 15 frequent event types that have the most number of instances in docee. docee covers a variety of event types, including armed conflict (8.0%), air crash (6.3%), protest (4.4%), commitcrime -accuse (4.1%), election (4.0%), sports competition (3.8%), fire (3.4%), etc. our annotated data follows a long-tailed distribution, which is due to the uneven distribution of class in the real data. according to statistics, there are 30.5% of classes with more than 500 instances and 83.1% of classes with more than 200 instances. more detailed information is shown in appendix ## experiments on docee in this section, we show the challenges of docee by conducting comprehensive experiments on various sota models. we first introduce two benchmark settings, and then we conduct experiments on both event classification task and event argument extraction task. finally, we discuss possible future research directions for document-level event extraction. ## conclusion in this paper, we present docee, a large-scale document-level ee dataset to promote event extraction from sentence-level to document-level. comparing to existing datasets, docee greatly expands the data scale, with more than 27,000+ events and 180,000+ arguments, and contains more refined event arguments. experiments show that docee remains an open issue.