Evaluating Zero-Shot Event Structures: Recommendations for Automatic Content Extraction ( ACE ) Annotations (2023)

Abstract: Zero-shot event extraction (EE) methods infer richly structured event records from text, based only on a minimal user specification and no training examples, which enables flexibility in exploring and developing applications. Most event extraction research uses the Automatic Content Extraction (ACE) annotated dataset to evaluate supervised EE methods, but can it be used to evaluate zero-shot and other low-supervision EE? We describe ACE’s event structures and identify significant ambiguities and issues in current evaluation practice, including (1) coreferent argument mentions, (2) conflicting argument head conventions, and (3) ignorance of modality and event class details. By sometimes mishandling these subtleties, current work may dramatically understate the actual performance of zero-shot and other low-supervision EE, considering up to 32% of correctly identified arguments and 25% of correctly ignored event mentions as false negatives. For each issue, we propose recommendations for future evaluations so the research community can better utilize ACE as an event evaluation resource.

## introduction zero-shot event extraction (ee) methods infer richly structured instances of action or relationship occurrences from unstructured text data, based on a user-supplied natural language specification of the desired event-without annotated training examples @xcite @xcite . the extracted structure is useful for many applications such as analyzing interactions between entities and performing more intelligent question answering @xcite @xcite , and the low resources required by zero-shot ee methods further this practical advantage. we refer to the structure as an event, where each event could have an arbitrary structure as needed. each struc-ture contains information such as the participants involved, content, and location of the event. to evaluate supervised ee methods, many works use the automatic content extraction (ace) dataset-specifically, the linguistic data consortium's ace 2005 multilingual training corpus @xcite ), foot_0 which includes english, chinese, and arabic documents and resulted from the u.s. federal government's ace program. 2 the ace dataset stores information about entities, relations, and events from 598 (for english) documents in a rich structure; our focus is mostly on its events. ace is frequently used for event extraction modeling and evaluation, and is often claimed to be the most widely used such dataset ( §3). while there are many somewhat similar structured semantic datasets, ace still shines in having whole-document annotations (contra framenet; @xcite @xcite , realistically non-lexical-specific event classes (contra propbank @xcite , ontonotes @xcite , and semantic dependencies @xcite ), event modality (contra pb, on, sd), english data (contra entities, relations, and events (ere)), foot_2 and specification of event arguments (contra richer event description (red); @xcite that are simultaneously represented both as text spans (contra abstract meaning representation (amr); @xcite , and discourse-level entities 4 ( §2). while ace does not include red's interesting causal and bridging event-event relations (see also @xcite , its core tasks related to entities and event arguments have important applications and are far from solved. we investigate using the ace dataset to evaluate zero-shot and other low-supervision ee methods, which are more real-world relevant than highlysupervised ee methods for requiring few if any annotations, but which may face certain evaluation challenges more severely. 5 first, we identify issues related to how evaluations extract gold event argument annotations from ace and to the possibly clashing use case of a zero-shot ee method versus the annotations in ace. evaluation of zero-shot ee methods is particularly sensitive to these issues since they lack knowledge of (sometimes arbitrary) details in ace event structures that are implicit in training examples-and their ignorance of them may be correct for many applications. therefore, we present guidelines and methods to overcome these issues in english, which could in theory be adaptable to other languages, and quantify their potential impact. 6 ## structure of events and entities in ace the automatic content extraction (ace) dataset stores annotations for entity, relation, and event structures for news, conversations, blog, and transcript textual data. we focus on the ace event extraction task @xcite , which takes a sentence as input and outputs a set of event tuples, which we attempt to precisely specify. events (figure 1 ). every event takes one of 33 discrete event classes @xmath0, 7 each of which has a discrete set of (typically, 1-6) roles r t which its 4 on 5.0 does include joint coreference and propbank annotations, albeit for only 18% of its english documents. entities (figure 2 ). an event argument a (s) may also be a mention of an entity, a document-level object with its own type information and one or more coreferential mention spans throughout a document. for an argument span a (s) , let c(a (s) ) refer to the set of all its coreferential mentions. 10 additionally, ace'@xmath1.2). in the following example from ace, a killing (life.die) event has agent a (s) ="iraq's mukhabarat" (figure 1 ); when cross-referencing the entity information c(a (s) ), it turns out this argument is coreferentially mentioned three times in the sentence (figure 2 ). earlier, from 1979 to 1983, he headed iraq's mukhabarat, or intelligence service, a period when the organization arranged executions of regime opponents in iraq and overseas, the official said. event extent: 2106:2264 type: life subtype: die trigger extent: 2214:2223 "executions" arguments (5) argument role: agent extent: 2130:2146 "iraq's mukhabarat" argument role: victim extent: 2228:2264 "regime opponents in iraq and overseas" argument role: place extent: 2248:2251 "iraq" argument role: time-starting extent: 2106:2109 "1979" argument role: time-ending extent: 2114:2117 "1983" 2106 2264 figure 1: event tuple in a sentence from ace document apw_eng_20030417.0555. earlier, from 1979 to 1983, he headed iraq's mukhabarat, or intelligence service, a period when the organization arranged executions of regime opponents in iraq and overseas, the official said. entity type: organization subtype: government mention extent: 2130:2146 "iraq's mukhabarat" mention extent: 2152:2171 "intelligence service" mention extent: 2188:2203 "the organization" head extent: 2137:2146 "mukhabarat" head extent: 2165:2171 "service" head extent: 2192:2203 "organization" ## review of using ace to evaluate ee we reviewed 38 papers published from 2008 through 2022, cited in li et al. (2022) 's survey of deep learning methods for event extraction, to examine how they use ace to evaluate ee tasks @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . several state that ace is the most popular dataset for evaluating ee methods @xcite @xcite . while the ace data release does not define a split, these papers, especially after 2011, settled on a shared train/development/test split ( §a.6). when considering event trigger and event argument identification, all papers require matching the gold standard's extent to be considered correct. for arguments, which are usually multiple tokens long, some works require matching the full argument extent a (s) while others only use its head extent. (additional details in §a.6.) the works that we analyzed identify several challenges with using ace. some event subtypes are very sparse; almost 60% of event types have fewer than 100 labeled samples, while three event types each have fewer than ten out of the 5042 samples over all english documents and 33 event classes @xcite @xcite . second, the manually specified event schemas in ace are hard to generalize to different domains, such as the weapon argument role @xcite . @xcite find that human annotators achieve only about 73% of the f1 score on the argument and trigger identification task and annotation quality continues to be questioned in debates about annotation guidelines @xcite . in any case, ace remains a widely used dataset for evaluation. ## recommendations for using ace to evaluate zero-shot and other low-supervision ee methods recommendation 1: coreference invariant argument matching. to evaluate correctness of event arguments using ace, allow a match to any coreferent mention of the argument (@xmath2, not just the one mention @xmath3. this (we believe) erroneous practice is widespread, and may consider up to 32% of correctly identified entitytype arguments as incorrect. problem. although ace stores event triggers and types as part of an event mention, it stores event arguments as part of both event mentions and entity, time, or value mentions. the event mention argument stores one reference to the argument (a (s) ), even if multiple references exist (c(a (s) )). low supervision ee methods can not learn a training set's potentially superficial convention for which of multiple references to specify. issues in the literature. alarmingly, although ace stores multiple gold references as entity mentions, they are often not used. we find that a number of recent works, especially on zero-shot ee, that ignore them. @xcite 's preprocessing code, which was used in several later works @xcite @xcite @xcite , does not gather multiple references to a (s) in an event tuple. while an unofficial update includes entity information, we identify further difficulties in §a.3. independently, @xcite acknowledge not applying coreference resolution, which contributes to a higher argument identification task error rate. while we acknowledge that whether to model coreference is a complex question, using gold standard coreference information at evaluation time is an independent issue and ought to be mandatory, for any modeling approach. even for a purely extent prediction system, gold-standard coreference is necessary for correct evaluation. findings. ## conclusion we explore how to use ace, which is a gold standard dataset containing annotations of events from diverse text data in a rich structure, to evaluate zero-shot and other low-supervision ee methods by identifying issues that may more severely affect their evaluation. we particularly find difficulties with evaluating spans of events due to a lack of training data for zero-shot and low-supervision ee methods to learn superficial annotation quirks from. however, we present methods to overcome these issues and demonstrate them on the english portion of ace, noting that in principle they may be adaptable to any language. ultimately, we advocate for using ace to evaluate zero-shot and other lowsupervision ee methods after addressing the issues, and discuss the potential for using ace in smarter ways to evaluate different types of ee methods in the future. li gao, jia wu, zhi qiao, chuan zhou, hong yang, and yue hu. 2016. collaborative social group influence for event recommendation. in proceedings of the 25th acm international on conference on information and knowledge management, cikm '16, page 1941-1944, new york, ny, usa. association for computing machinery. salvatore giorgi, vanni zavarella, hristo tanev, nicolas stefanovitch, sy hwang, hansi hettiarachchi, tharindu ranasinghe, vivek kalyan, paul tan, shaun tan, martin andrews, tiancheng hu, niklas stoehr, francesco ignazio re, daniel vegh, dennis atzenhofer, brenda curtis, and ali hürriyetoglu. 2021. discovering black lives matter events in the united states: shared task 3, case 2021. in proceedings of the 4th workshop on challenges and applications of automated extraction of socio-political events from text (case 2021), pages 218-227, online. association for computational linguistics. aria haghighi and dan klein. 2009. simple coreference resolution with rich syntactic and semantic features. in proceedings of the 2009 conference on empirical methods in natural language processing, pages 1152-1161, singapore. association for computational linguistics. andrew halterman, katherine keith, sheikh sarwar, and brendan o'connor. 2021. corpus-level evaluation for event qa: the indiapoliceevents corpus covering the 2002 gujarat violence. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 4240-4253, online. association for computational linguistics. alex hanna. 2017. mpeds: automating the generation of protest event data. yu hong, jianfeng zhang, bin ma, jianmin yao, guodong zhou, and qiaoming zhu. 2011. using cross-entity inference to improve event extraction. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 1127-1136, portland, oregon, usa. association for computational linguistics. yu hong, wenxuan zhou, jingli zhang, guodong zhou, and qiaoming zhu. 2018. self-regulation: employing a generative adversarial network to improve event detection. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 515-526, melbourne, australia. association for computational linguistics. eduard hovy, teruko mitamura, felisa verdejo, jun araki, and andrew philpot. 2013. events are not simple: identity, non-identity, and quasi-identity. in workshop on events: definition, detection, coreference, and representation, pages 21-28, atlanta, georgia. association for computational linguistics. lifu huang, taylor cassidy, xiaocheng feng, heng ji, clare r. voss, jiawei han, and avirup sil. 2016. liberal event extraction and event schema induction. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 258-268, berlin, germany. association for computational linguistics. lifu huang, heng ji, kyunghyun cho, ido dagan, sebastian riedel, and clare voss. 2018. zero-shot transfer learning for event extraction. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2160-2170, melbourne, australia. association for computational linguistics. ali hürriyetoglu, osman mutlu, erdem yörük, farhana ferdousi liza, ritesh kumar, and shyam ratan. 2021. multilingual protest news detectionshared task 1, case 2021. in proceedings of the 4th workshop on challenges and applications of automated extraction of socio-political events from text (case 2021), pages 79-91, online. association for computational linguistics. heng ji and ralph grishman. 2008. refining event extraction through cross-document inference. in proceedings of acl-08: hlt, pages 254-262, columbus, ohio. association for computational linguistics. kenton lee, yoav artzi, yejin choi, and luke zettlemoyer. 2015. event detection and factuality assessment with non-expert supervision. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 1643-1648, lisbon, portugal. association for computational linguistics. fayuan li, weihua peng, yuguang chen, quan wang, lu pan, yajuan lyu, and yong zhu. 2020a. event extraction as multi-turn question answering. in findings of the association for computational linguistics: emnlp 2020, pages 829-838, online. association for computational linguistics. manling li, alireza zareian, ying lin, xiaoman pan, spencer whitehead, brian chen, bo wu, heng ji, shih-fu chang, clare voss, daniel napierski, and marjorie freedman. 2020b. gaia: a fine-grained multimedia knowledge extraction system. in proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations, pages 77-86, online. association for computational linguistics. qi li, heng ji, and liang huang. 2013. joint event extraction via structured prediction with global features. in proceedings of the 51st annual meeting of the association for computational linguistics (volume 1: long papers), pages 73-82, sofia, bulgaria. association for computational linguistics. qian li, hao peng, jianxin li, yiming hei, rui sun, jiawei sheng, shu guo, lihong wang, and philip s. yu. 2022. a survey on deep learning event extraction: approaches and applications. ieee transactions on neural networks and learning systems. short papers), pages 414-419, melbourne, australia. association for computational linguistics. yang zhou, yubo chen, jun zhao, yin wu, jiexin xu, and jinlong li. 2021. what the role is vs. what plays the role: semi-supervised event argument extraction via dual question answering. in aaai conference on artificial intelligence.