DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction
Abstract
Event extraction aims to identify an event and then extract the arguments participating in the event.
Despite the great success in sentencelevel event extraction, events are more naturally presented in the form of documents, with event arguments scattered in multiple sentences.
However, a major barrier to promote documentlevel event extraction has been the lack of large-scale and practical training and evaluation datasets.
In this paper, we present DocEE, a new document-level event extraction dataset including 27,000+ events, 180,000+ arguments.
We highlight three features: largescale manual annotations, fine-grained argument types and application-oriented settings.
Experiments show that there is still a big gap between state-of-the-art models and human beings (41% Vs 85% in F1 score), indicating that DocEE is an open issue.
DocEE is now available at https://github.com/ tongmeihan DocEE git .
Introduction
Event Extraction (EE) aims to detect events from text, including event classification and event argument extraction.
EE is one of the fundamental tasks in text mining (Feldman and Sanger, 2006) and has many applications.
For instance, it can monitor political or military crises to generate real-time notifications and alerts (Dragos, 2013), and dig the links and connections (e.g., Who Met Whom and When) between dignitaries for portrait analysis (Zhan et al., 2020).
Most existing datasets (e.g., ACE2005 1 and
KBP2017 2 ) focus on sentence-level event extraction, while events are usually described at the document level, and event arguments are typically scattered across different sentences (Hamborg et al.,
Figure 1 shows an Air Crash event.
To extract argument Date , we need to read sentence [1], while to extract argument Cause of the Accident , we need to integrate information in sentences [6] and [7].
Clearly, this requires reasoning over multiple sentences and modeling long-distance dependency, intuitively beyond the reach of sentence-level EE.
Therefore, it is necessary to move EE forward from sentence-level to document-level.
Only a few datasets are curated for documentlevel EE.
MUC-4(Grishman and Sundheim, 1996) provides 1,700 news articles annotated with 4 event types and 5 argument types.
The 5 arguments are shared among different event types without further refinement.
WikiEvents(Li et al., 2021) consists of only 246 documents with very few (22% of total) cross-sentences argument annotations.
RAMS(Ebner et al., 2020) limits the scope of the arguments in a 5-sentence window around its event trigger, which is not in line with the actual application, and the number of the argument types in RAMS is only 65, which is quite limited.
Doc2EDAG, TDJEE and GIT (Zheng et al., 2019;
Wang et al., 2021; Xu et al., 2021) contain onlyevent types and 35 argument types in financial domain.
In summary, existing datasets for documentlevel EE fail in the following aspects: small scale of data, limited coverage of domain and insufficient refinement of argument types.
Therefore, it is urgent to develop a manually labeled, large-scale dataset to accelerate the research in document-level
EE.
In the paper, we present DocEE, a large-scale human-annotated document-level EE dataset.
Figure 1 illustrates an example of DocEE.
DocEE focus on the extraction of the main event, that is oneevent-per-document .
We regard news headlines as the main event trigger and focus on main event arguments extraction throughout the article.
We highlight the following three contributions of DocEE to this
field: 1) Large-scale Manual Annotations.
Do-
Date
Location
Casualties and Losses
Survivors
Investigator of the Accident
Cause of the Accident
Passengers
Crew
Aircraft Agency
Flight No.
Service years
Taking-off Place
Scheduled Landing Place
Alternate Landing Place
NAF Plane Crash: Military Takes Over Site, AIB To Commence Investigation
Event Type: Air Crash
Event Roles [1]Soldiers have cordoned off the site where the Nigerian Airforce plane crashed on Friday evening at the Kaduna International Airport.
[2]The plane which was carrying the Chief of Army Staff, Lieutenant General Ibrahim Attahiru, and other Senior Army Ofﬁcers crashed near the airport’s active runway, killing all the eleven people on board.
[3]Speaking on the incident, the Manager of Kaduna airport, Amina Salami told Channels Television that the military authorities have taken over the crash site.
[4]She added that they have prevented civil aviation ofﬁcials from gaining access to the crash site pending the arrival of ofﬁcials from the Accident Investigation Bureau (AIB).
[5]She explained that the military plane was initially scheduled to land at the
Nigerian Airforce Base in Mando area, but was later diverted to the Kaduna airport due to poor weather conditions.
The service life of the aircraft is three years.
[6]The incident happened as the plane was trying to land in bad weather, the military said.
[7]President Muhammadu Buhari said that the lack of strict pre-ﬂight inspections was also the cause of this disaster.
[8]It was part of the government's plan to boost the military's efﬁciency in ﬁghting a more than decade- long jihadist insurgency.
cEE contains 27,485 document-level events with 180,528 arguments, far exceeding the scale of existing document-level EE datasets. The large-scale annotations of DocEE can provide sufficient training and testing data, to fairly evaluate EE models.
2) Fine-grained argument types.
DocEE has a total of 356 argument types, which is much more than the number of argument types in existing dataset ( in MUC-5 and 65 in RAMS).
Besides the general arguments, such as time and location, we design more personalized event arguments for each event type, such as Water Level for Flood event and Magnitude for Earthquake event.
These fine-grained roles can bring more detailed semantics and pose a higher challenge to the semantic disambiguation ability of existing models.
3) Application-oriented settings.
In the actual application, event extraction often face the problems of how to quickly adapt from the rich-resource domains to new domains.
Therefore, we have added a cross-domain setting to better test the transfer capability of the EE models.
In addition, DocEE removes the limitation that the arguments range should be within a certain window in RAMS, to better cope with realistic scenarios where the length of the article will be particularly long, and the argument of the event may appear in any corner of the article.
With more scattered event arguments (see Table 1), DocEE poses a higher challenge to the long text processing capability of existing models.
To assess the challenges of DocEE, we implement 9 recent state-of-the-art EE models on
DocEE along with human evaluation.
Experiments demonstrate the high-quality of DocEE and show that even the performance of SOTA model is far lower than human performance, showing that the faintness of existing technology in processing document-level EE.
Related Datasets
Sentence-level Event Extraction Dataset Automatic Content Extraction (ACE2005) 1 consists of 599 documents with 8 event types and 33 subtypes.
Text Analysis Conference (TACKBP) 2 also releases three benchmarks: TACKBP 2015/2016/2017, with 9/8/8 event types and 38/18/18 event subtypes.
RED 3 annotates events from 95 English newswires.
Chinese Emergency
Corpus (CEC) focuses on Chinese breaking news, with a total of 332 articles in 5 categories.
MAVEN (Wang et al., 2020) and LSEE (Chen et al., 2017) only annotate event triggers, with 168/21 types of trigger instances in 11,832/72,611 sentences.
Based on them, various superior models have been proposed to improve the sentence-level EE and have achieved great success (Orr et al., 2018;
Nguyen and Grishman, 2018; Tong et al., 2020).
Document-level Event Extraction Dataset Most of the existing document-level event datasets only focus on event classification, but lack event argument labelings, such as 20news 4 and THUCNews 5 .
There are a few datasets annotated with cross-sentences event arguments.
MUC-4 (Nguyen
[Figure]
Spacecraft Launch Train Collision Flood
Sports Competition
Protest • Date • Location • Launch Country • Astronauts • Spacecraft • Carrier rocket • Spacecraft Mission • Research Agency • Cooperative Agency • Spokesman • Launch Result • Mission duration • Date • Location • Train Agency • Survivors • Wounded Hospital •
Casualties and Losses • Accident Investigator • Cause •
Economic Loss • Train
No.
• Missing • Date • Areas Affected • Water Level • Maximum Rainfall • Related Rivers • Casualties and
Losses • Cause • Aid Agency& Supplies •
Temporary Settlement • Economic Loss • Missings • Damaged Farmland • Damaged Houses • Number of Evacuated • Number of Rescued •
Start Time •
End Time • Lasting time • Location • Host country • Contest participant • Champions • MVP • Score • Game Name • Competition items • Postpone Reason • Postpone Time • Date • Location • Protesters • Cause • Slogan • Method • Arrested • Government Reaction • Casualties and Losses • Damaged Property et al., 2016) only contains 4 event types and 5 argument types, and the 4 event types are close to each other and limited to the terrorist attack topic 6 .
WikiEvents (Li et al., 2021) and RAMS(Ebner et al., 2020) consist of 246/9,124 documents with only 59/65 argument types, and most of the arguments in the two datasets are shared among different event types without further refinement.
Doc2EDAG, TDJEE and GIT (Zheng et al., 2019;
Wang et al., 2021; Xu et al., 2021) only define 5 event types and 35 argument types in financial domain.
Cancer Genetics, EPM, GENIA2011, GENIA2013, Pathway Curation and MLEE (Pyysalo et al., 2013; Ohta et al., 2011; Kim et al., 2011, 2013; Ohta et al., 2013; Van Landeghem et al., 2013) are limited to the biological domain.
In summary, these datasets are either limited to specific domains, or have very limited data scale, or have not carefully refined event argument schema.
Open-domain Event Extraction Dataset To collect EE dataset in open domain, one way is to leverage semi-structured resources (Wikipedia) or existing knowledge bases (Freebase). The representative works are EventKG (Gottschalk and Demidova, 2018), Event Wiki (Ge et al., 2018) and Historical Wiki (Hienert and Luciano, 2012).
The other way is to exploit open IE tools, such as dependency parsing, to extract events from unstructured text.
The representative works are Event Logic Graph (Ding et al., 2019) and
Giveme5W1H (Hamborg et al., 2019).
The advantage of the open-domain
EE dataset is its large scale, but the disadvantage is that it lacks manual review, and thus the quality cannot be guaranteed.
Constructing DocEE
Our main goal is to collect a large-scale dataset to promote the development of event extraction from sentence-level to document-level.
In the following sections, we will first introduce how to construct the event schema, and then how to collect candidate data and how to label them through crowdsourcing.
3. Event Schema Construction
News is the first-hand source of hot events, so we focus on extracting events from news.
Previous event schemas, such as FrameNet (Baker, 2014) and HowNet (Dong and Dong, 2003), pay more attention to trivial actions such as eating and sleeping , and thus is not suitable for document-level news event extraction.
To construct event schema, we gain insight from journalism.
Journalism typically divides events into hard news and soft news (Reinemann et al., 2012; Tuchman, 1973).
Hard news is a social emergency that must be reported immediately, such as earthquakes, road accidents and armed conflicts.
Soft news refers to interesting incidents related to human life, such as celebrity deeds, sports events and other entertainment-centric reports.
Based on the hard/soft news theory and the category framework in (Lehman-Wilzig and Seletzky, 2010), we define a total of 59 event types, with 31 hard news event types and 28 soft news event types.
Detailed information is shown in Appendix Table 1.
Our schema covers influential events of human concern, such as earthquakes, floods and diplomatic summits, which cannot be extracted at the sentence level and require multiple sentences to describe.
To construct argument schema, we leverage infobox in Wikipedia.
As shown in Figure 3(a), the
Wikipedia page describes an event, and the keys in the infobox, such as Date and Total fatalities , can be regarded as the prototype arguments of the event.
Based on this observation, we manually collect 20 wiki pages for each event type, and use their shared keys in infobox as our basic set of argument types.
After that, we further expand the basic set.
Specifically, for event type e , we first collectnews stories from New York Times, and then invited 5 students (native English-speaking, major in journalism) to summarize the key facts the public would like to learn from the news of e .
For instance, in Flood event news, Water Level is a key fact, because it is an important factual basis for flood cause analysis and disaster relief decision-making, and can arouse widespread concern.
Finally, by merging the key facts of the 5 students, we complete the argument types expansion.
To ensure the quality, we further invite the above 5 students to make a trial labeling on the collected news, and filter argument types that appear less frequently in the article.
In total, we define 356 event arguments types for 59 event types.
On average, there are 6.0 event arguments per class.
Figure 2 illiterates some examples of event arguments types we defined.
The complete schema and corresponding examples can be found Event Schema.md in the supplementary materials.
3. Candidate Data Collection
In this section, we introduce how to collect candidate document-level events.
We choose wiki as our annotation source.
Wiki contains two kinds of events: historical events and timeline events (Hienert and Luciano, 2012).
Historical events refer to the events that have their own wiki page, such as 1922 Picardie mid-air collision .
Timeline events refer to the news events organized in chronological order, such as A heatwave strikes India and South Asia in wiki page Portal:Current_events/June_2010 .
7 Figure 3 shows examples of two events.
We adopt both kinds of events as our candidate data, because only using historical events will lead to uneven data distribution under our event schema, and timeline events can be a good supplement.
For a historical event, we adopt its Wikipedia article as the document of the event arguments to be annotated.
For a timeline event, we use the
URL to download the original news article as the document of the event arguments to be annotated.
Because 22% of the timeline events do not have
URLs (Wikipedia editors do not provide the URL when editing the entry), so we use Scale SERPto find news articles and manually confirm their authenticity.
For historical event, we adopt templates+event type as the query key to retrieve candidate events.
The templates includes "List of"+event type , event type+"in"+year , "Category:"+event type+"in"+country , etc. More templates show in
Appendix Table 7. For timeline event, we choose events between 1980 and 2021 as candidates, because there are very few events before 1980.
In order to balance the length of the article, we filtered out articles less than 5 sentences, and also truncated articles that were too long (more than 50 sentences).
Finally, we select 44,000 candidate events from Wikipedia.
3. Crowdsourced Labeling
Given the candidate events and the predefined event schema, we now introduce how to annotate them through crowdsourcing.
To ensure the quality of annotations, all annotators are either native
English speakers or English-major students with
TOEFL higher than 100 or IELTS higher than 7.5.
The crowdsourced labeling process consists of two stages.
3.3. Stage 1: Event Classification
At this stage, annotators are required to classify candidate events into predefined event types.
Following (Hamborg et al., 2018; Hsi, 2018), we focus on main event classification, so Stage 1 is a single-label classification task.
Specifically, the main event refers to the event reflected in the title and mainly described in the article.
Formally, given the candidate event e = < t, a > , where t represents the title and a represents the article, Stageaims to obtain label y for each e , where y belongs to the 59 event types defined in subsection 3.1.
In total, we invite about 60 annotators to participate in Stage 1 annotation.
The online annotation page is displayed in Appendix Figure 5.
We first manually label 100 articles as standard answers to pre-test annotators, and weed out annotators with an accuracy rate of less than 70%, which left us 48 valid annotators.
Then, we ask two independent annotators to annotate each candidate event.
If the results of the two annotators are inconsistent [Figure] (32.8% in this case), a third annotator will be the final judge.
Due to the variety of event types in reality, a candidate event may not belong to any predefined class.
We classify such event into the other class, which accounts for 23.6% of the total data.
3.3. Stage 2: Event argument Extraction
At this stage, annotators are required to extract event arguments from the whole article.
Formally, given the candidate event e = < t, a > , its event type y and the predefined argument types R of y ,
Stage 2 aims to find all the arguments from the article a .
Due to the heavy workload in Stage 2, we invite more than 90 annotators.
An example of the online annotation page is shown in Appendix Figure 6.
We use a preliminary annotation - multiple rounds inspection method for labeling.
In the preliminary annotation step, each article will be labeled by an annotator.
We distribute no more than two event types to each annotator in this step to make the annotators more focused.
Then, in the step of multiple rounds inspection, we first select high-precision annotators based on inter-annotator agreement to form a reviewer team (44.4% of the total), and then each article will go through three rounds of error correction by three independent annotators in the reviewer team.
After each round, we will feed back annotation issues to the reviewers so that they can correct them in the next round of annotation.
The accuracy rate has steadily increased from 56.24%, 76.83% to 85.96% after each round, which shows the effectiveness of our labeling method.
We take the third round results as the final annotations.
We clarify some annotation details here.
We do not include articles, prepositions in our annotations.
For instance, we select "damaged car" among "damaged car", "damaged car belonging to the victim" and "the damaged car".
For event arguments with multiple mentions in the document, for example,
Cause of the Accident in Figure 1 that has two mentions, we will label all mentions to ensure the completeness of the extraction.
For repeated mentions that refer to the same entity, we only label once.
3.3. Annotation Quality & Remuneration
Following (Artstein and Poesio, 2008; McHugh, 2012), we use Cohen’s kappa coefficient to measure the Inter-Annotator Agreement(IAA).
The
IAA scores are 94% and 81% for State 1 Event
Classification and Stage 2 Event Argument Extraction respectively, which are relatively high.
The annotators spend an average of 0.5 minutes labeling a piece of data in Stage 1, so we pay them 0.1$ for each piece of data.
It takes about 5 minutes to label a piece of data in Stage 2 , so we pay 0.8$ for each piece of data.
Data Analysis of DocEE
In the section, we analyze various aspects of DocEE to provide a deep understanding of the dataset and the task of document-level event extraction.
Overall Statistic
In total, DocEE labels 27, valid document-level events and 180,528 event arguments.
Each article is annotated with 6.6 event arguments on average.
Event Famous PersonDivorce has the highest average number of event arguments per article (18.1), while event Regime
Change has the lowest average number of event arguments per article (3.8).
We compare DocEE to various representative event extraction datasets in Table 1, including sentence-level EE datasets
Datasets #isDocEvent #EvTyp.
#ArgTyp.
#Doc.
#Tok.
#Sent.
#ArgInst.
#ArgScat.
ACE ✗   290k 15, 9, 1 KBP ✗   94k 5, 7, 1 KBP ✗   86k 4, 10, 1 MUC-
[Figure]
✓  1, 495k 21, 2, 4.0 WikiEvents ✓   190k 8, 5, 2.2 RAMS ✓  9, 957k 34, 21, 4. DocEE(ours) ✓  27, 16,268k 749, 180, 10.
[Figure]
ACE2005, KBP and document-level EE dataset MUC-4, Wikievents, RAMS. We find that DocEE is much larger than existing datasets in many aspects, including the documents number and argument instances number. Compared to MUC-4, DocEE has far more event arguments (180,528 compared to 2,641).
The reason is that among the 1, documents in MUC-4, 47.4% of articles are not labeled with any event argument, while DocEE guarantees that each article contains at least three event argument labels in crowdsourcing process, which greatly solves the problem of data scarcity of the event arguments in document-level EE.
Event Type Statistic Figure 4 shows the distribution of the top 15 frequent event types that have the most number of instances in DocEE.
DocEE covers a variety of event types, including Armed
Conflict (8.0%), Air crash (6.3%), Protest (4.4%),
CommitCrime - Accuse (4.1%), Election (4.0%),
Sports Competition (3.8%), Fire (3.4%), etc.
Our annotated data follows a long-tailed distribution, which is due to the uneven distribution of class in the real data.
According to statistics, there are 30.5% of classes with more than 500 instances and 83.1% of classes with more than 200 instances.
More detailed information is shown in Appendix
Table
Event Arguments Statistic We randomly sample 1000 articles from DocEE for manual analysis, which contains a total of 4962 event arguments instances.
We first classify event arguments based on their mention numbers.
As shown in Table 2, 78.6% of the event arguments have a unique mention, and 21.4% of the event arguments have multiple mentions, which poses a great challenge to the model’s recall capability.
Then, we classify event arguments based on their mention lengths. 60.8% of the event arguments have no more than 3 words, and most of them are named entities such as person and location.
While 30.8% event arguments have less than 10 words and 8.4% event arguments are answered by more than 10 words, such event arguments mainly include Cause of the Accident ,
Investigation Results , etc.
Experiments on DocEE
In this section, we show the challenges of DocEE by conducting comprehensive experiments on various SOTA models.
We first introduce two benchmark settings, and then we conduct experiments on both event classification task and event argument extraction task.
Finally, we discuss possible future research directions for document-level event extraction.
Benchmark Settings We design two benchmark settings for evaluation: normal setting and crossdomain setting.
In the normal setting, we hope the training set and test set to be identically distributed.
Specifically, for each event type, we randomly select 80% of the data as the training set, 10% of the data as the validation set, and the remaining 10% of the data as the test set. [Table]
In order to be application-oriented, we design cross-domain setting to test the transfer capability of the SOTA models.
We choose the event type under the subject of natural disasters as the target domain, including Floods, Droughts, Earthquakes,
Insect Disaster, Famine, Tsunamis, Mudslides, Hurricanes, Fire and Volcano Eruption [Table] and adopt the remaining 49 event types as source domains.
The division reduces the overlap of argument types between the source domain and the target domain.
In this setting, the models will first be pre-trained on the source domain [Table] and then conduct 5-shot finetuning on the target domain.
The detailed data split for each setting is shown in Table [Table]
Hyperparameters We use base version of pretrained model for all the transformer-based methods, and set the learning rate to 2e-5.
The batch size is 128 and the maximum document length is 512.
All baselines are implemented by HuggingFacewith default parameters and all models can be fit into eight V100 GPUs with 16G memory.
The training procedure lasts for about a few hours.
For all the experiments, we report the average result of five runs as the final result.
In human evaluation, we randomly select 1,000 document-level events and invite three students to label them.
The final result is the average of their labeling accuracy.
5. Event Classification
Baselines We adopt a CNN-based method and various pre-trained transformer-based methods as our baselines, including: 1) TextCNN (Kim, 2014) uses different sizes CNN kernels to extract key information in text for classification.
2) BERT (Devlin et al., 2018) exploits unsupervised objective functions, masking language model (MLM) and next sentence prediction for pre-training.
3) ALBERT (Lan et al., 2020) proposes a self-supervised loss to improve inter-sentence coherence in BERT.
4) DistilBert (Sanh et al., 2019) combines language modeling, knowledge distillation and cosinedistance losses to improve BERT.
5) RoBERTa (Liu et al., 2019) is built on BERT and trains with much larger mini-batches and learning rates.
Following (Kowsari et al., 2019), we use Precision(P),
Recall(R) and Macro-F1 score as the evaluation metrics.
[Table]
Overall Performance Table 4 shows the experimental results under the normal and cross-domain settings, from which we have the following observations: 1) Compared with TextCNN, transformer based models (BERT, ALBERT, DistillBert,
RoBERTa) perform better, which are pre-trained on a large-scale unsupervised corpus and have more background semantic knowledge to rely on. 2)
Humans have achieved high scores on DocEE, verifying the high quality of our annotated data sets.
There is still a gap between the performance of the current SOTA models and human beings, which indicates that more technological advances are needed in future work.
Humans can connect and merge key information to form a knowledge [Table] network to help them understand the main event, while deep learning models typically fail in long text perception.
There is a performance degradation from the normal setting to the cross-domain setting, which shows that domain migration is still a huge challenge for current SOTA models.
Among the pre-trained baselines, ALBERT’s performance drops the most.
The reason may be that the parameter scale in ALBERT is relatively small, and the reserved source domain knowledge is limited.
5. Event argument Extraction
Baselines We introduce the following mainstream baselines for evaluation: 1) BERT_Seq (one of the baseline in Du and Cardie (2020a)) uses the pre-trained BERT model to sequentially label words in the article.
Given the input article A = { w 1 , w 2 , . . . , w n } , the output of Sequence Labeling Methods is O = { r 1 , r 2 , . . .
, r n } , where r ∈ R and R is the set of the argument types.
2) MG-Reader (Du and Cardie, 2020a) improves document-level EE by proposing a novel multigranularity reader to dynamically aggregate information in sentence and paragraph-level. 3) DocEDAG (Zheng et al., 2019) generates an entitybased directed acyclic graph for document-level
EE. 4) BERT_QA (Du and Cardie, 2020c) uses the argument type as question to query the article for answer.
Given the input article A , the argument type r ∈ R as the question, the output is O = { start r , end r } .
We give − 1 for these not mentioned event arguments.
5) Ontology_QA .
Following Vargas-Vera and Motta (2004), we refine the initial query in BERT_QA with argument ontology knowledge obtained from Oxford dictionary (Dictionary, 1989).
Considering the length limitation of pre-trained models, we split the article in three different ways.
(Sent) means to split the article by sentence 10 .
(Chunk) means to split the article by everytokens (default).
(Doc) means no splitting.
We adopt Longformer (Beltagy et al., 2020) as encoder for the (doc) baseline, and BERT-base for the other baselines.
Following prior work (Du and Cardie, 2020b), we use Head noun phrase Match (HM) and Exact
Match (EM) as two evaluation metrics.
HM is a relatively relaxed metric.
As long as the head noun of the predicted result is consistent with the golden label, it will be judged as correct.
While
EM requires that the prediction result is exactly the same as the gold label, which is relatively stricter.
Overall Performance As shown in Table 5, there is a big gap between the performance of SOTA models and human performance (41.0% Vs 85.9% in F score), indicating that document-level event argument extraction remains a challenge task.
The failure of existing baselines may be due to two reasons.
One possible reason is the catastrophic forgetting in neural networks.
Compared to NER and sentence-level EE, document-level EE (our task) highlights the model’s capability to process long texts: the model has to read the entire text before determining the argument type of a span.
Although a few models have been proposed to improve the long text capabilities of pre-trained models (such as longformer), and have achieved good results, (the performance of long-former (BERT_Seq(doc)) is superior to BERT_Seq(sent) and BERT_Seq(chunk) as shown in Table 5), but these models still have a big performance gap compared with human beings. Another reason is that existing baselines suffer from the inferior capability in semantic understanding, which is reflected in two aspects: 1) EE models fail to distinguish arguments of similar events. For instance, the article mainly describes the 2021 U.S. Alaska Peninsula earthquake , and also briefly mentions 2008 Wenchuan earthquake .
When asking the Date of the main event, EE models are easy to confuse the correct answer 2021 with the wrong answer 2008. 2) EE models often mistake unrelated entities for event arguments.
For example, when extracting the event argument Attack Target in the the 911 terrorist attack on the Pentagon event, except to the correct answer the New York Pentagon , EE models often mistake other unrelated location entities in the article (such as Mount Sinai Hospital ) as one of the answers.
We believe that the following research directions are worthy of attention: 1) Exploring pre-trained models with stronger long text processing capabilities.
Exploiting ontology and commonsense knowledge to improve the semantic understanding of EE models.
In the future, we will focus on promote event extraction to a higher level, such as cross-document level.
Conclusion
In this paper, we present DocEE, a large-scale document-level EE dataset to promote event extraction from sentence-level to document-level.
Comparing to existing datasets, DocEE greatly expands the data scale, with more than 27,000+ events and 180,000+ arguments, and contains more refined event arguments.
Experiments show that DocEE remains an open issue.
Acknowledgements
This work is supported by the National Key
Research and Development Program of China (2018YFB1005100) and (2018YFB1005101), National Engineering Laboratory for Cyberlearning and Intelligent Technology, Beijing Key Lab of Networked Multimedia, the Institute for Guo Qiang,
Tsinghua University (2019GQB0003), the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF)
Tier 1 grant, and Beijing
Academy of Artificial Intelligence.