LEMONADE : A Large Multilingual Expert-Annotated Abstractive Event Dataset for the Real World (2025)

Abstract: This paper presents LEMONADE, a large-scale conflict event dataset comprising 39,786 events across 20 languages and 171 countries, with extensive coverage of region-specific entities. LEMONADE is based on a partially reannotated subset of the Armed Conflict Location & Event Data (ACLED), which has documented global conflict events for over a decade.To address the challenge of aggregating multilingual sources for global event analysis, we introduce abstractive event extraction (AEE) and its subtask, abstractive entity linking (AEL). Unlike conventional span-based event extraction, our approach detects event arguments and entities through holistic document understanding and normalizes them across the multilingual dataset. We evaluate various large language models (LLMs) on these tasks, adapt existing zero-shot event extraction systems, and benchmark supervised models. Additionally, we introduce ZEST, a novel zero-shot retrieval-based system for AEL.Our best zero-shot system achieves an end-to-end F1 score of 58.3%, with LLMs outperforming specialized event extraction models such as GoLLIE. For entity linking, ZEST achieves an F1 score of 45.7%, significantly surpassing OneNet, a state-of-the-art zero-shot baseline that achieves only 23.7%. However, these zero-shot results lag behind the best supervised systems by 20.1% and 37.0% in the end-to-end and AEL tasks, respectively, highlighting the need for further research.

## introduction event extraction (ee) involves extracting structured information about events and their arguments from unstructured text, such as news articles. this task is fundamental for understanding and analyzing real-world phenomena at scale. this paper refines the ee task to better serve the study of global real-world phenomena. as a case study, we analyze data from the armed conflict location & event data (acled), a non-profit organization that has systematically documented violent conflict and protest events worldwide for over a decade @xcite . acled's data supports critical humanitarian work by organizations including the united nations' international organization for migration, the international rescue committee, and the european commission, who use it to track forced displacement and evaluate humanitarian interventions @xcite . based on this analysis, we introduce lemon-ade, a cleaned and partially reannotated version of the acled dataset tailored for nlp research. lemonade addresses several critical gaps in existing event extraction resources: multilinguality and geographic diversity to provide a truly global perspective, event extraction must extend beyond the global north to include perspectives from the global south and international regions @xcite . unlike existing ee datasets that focus primarily on english or chinese, lemonade encompasses events across 171 countries reported in 20 languages. tail entity coverage entity linking is essential for aggregating information about event participants. while general-purpose entity databases like wikidata @xcite and wikipedia @xcite offer broad coverage, they often lack specialized domain entities. lemonade addresses this gap with a database of 10,707 entities, including: hashemiyoun", a syrian political militia active since 2023) • regional organizations (e.g., "nno: nagorik nari oikya", the women's wing of nagarik oikya in bangladesh) many of these entities lack wikipedia entries, creating unique challenges for entity linking systems. this is particularly significant because most large language models (llms) are pre-trained on wikipedia and tend to memorize common entities. 2 expert annotations high-quality annotations are essential when ee systems inform high-stakes policy decisions, such as international peacemaking efforts @xcite , where annotation errors can lead to biased conclusions. while prior work @xcite has emphasized the importance of domain expertise, most existing document-level ee datasets rely on crowdsourcing @xcite @xcite , student annotators @xcite , or weakly supervised methods @xcite . in contrast, acled employs about 200 regional experts who conduct multiple review rounds, ensuring high annotation quality and consistency. our contributions are threefold: entity detection and linking. entity linking (el) connects entity mentions in text to entries in a database @xcite . traditional el pipelines first detect entity mention spans, then disambiguate them against the target database. in contrast, our abstractive entity linking (ael) approach directly maps input text to a list of linked entities, without explicit span detection. zero-shot el, which enables linking to new entity databases without direct supervision, is particularly relevant to our work. @xcite demonstrated the effectiveness of pre-trained models for zero-shot el and introduced zeshel, now a popular benchmark. recent advances include @xcite 's read-and- text input mobviolence( @xmath0, @xmath1, 'ex-sarpanch's husband and her family members' ] @xmath2, 'the lady sarpanch, pratibha mallick', ], ... ) lady sarpanch, husband 'attacked' by ex-sarpanch in odisha's jajpur bhubaneswar, jan. 19 --a lady sarpanch and her husband have been critically injured after they were allegedly attacked by a former sarpanch and supporters in chaskhand panchayat in jajpur district. the lady sarpanch, pratibha mallick, and her husband were thrashed by the ex-sarpanch's husband and her family members following a dispute over laying of pipeline under the basudha scheme. ... text input figure 1 : an example from lemonade showing abstractive event annotation. the input text and annotations are summarized for clarity. a hypothetical extractive annotation is included for comparison, illustrating the key differences between abstractive and extractive approaches. ## the abstractive event extraction task the aee task shifts focus from surface text forms to grounding events in predefined ontologies and representing arguments as categorical, numerical, or string values. aee removes two key constraints compared to traditional event extraction: arguments need not be text spans, and they need not be explicitly mentioned in the text. definition 3.1. an event extraction codebook @xmath3, d, s) consists of: definition 3.2. the abstractive event extrac-tion (aee) task is: given codebook @xmath4, d, s) and text w, extract abstractive event(s) of the form (t i , v 1 , . . . , v n i ) from w, where t @xmath5, each v @xmath6,j is a value for argument a j , and n i is the number of arguments for event type t i . figure 1 demonstrates how aee can, for example, facilitate studying violence against women globally. for event type t @xmath7, its first two arguments, perpetrators and victims represent the two sides in the violence, with d i,1 , d i,2 being the set of all subsets of possible entities from the event database. the seventh argument targets_women is a boolean, thus d i,7 is {true, false}. the aee annotation captures critical information not explicitly stated as text spans in the input: these abstract arguments-represented as boolean, enum, and numerical fields-enable straightforward aggregation for analytical queries such as "how many casualties resulted from violence against female government officials in 2024?" aee eliminates the need for intermediate annotations such as event triggers and entity mentions @xcite , avoiding the common practice of annotating multiple spans for the same argument. instead, aee directly annotates events and their linked entities, enabling direct evaluation against gold annotations. this streamlined approach reduces annotation complexity, produces cleaner labels, and allows simple exact-match evaluation, addressing limitations of existing ee metrics @xcite . the aee framework comprises three core subtasks: ed(w, c) = {t 1 , . . .} ⊆ t an end-to-end aee system first performs ed, then uses predicted event types (rather than gold types) to separately perform aeae and ael. ## lemonade: an aee dataset lemonade spans 20 typologically diverse languages @xcite , ordered by the number of examples in lemonade from most to least: english, spanish, arabic, french, italian, russian, german, turkish, burmese, indonesian, ukrainian, korean, portuguese, dutch, somali, nepali, chinese, persian, hebrew, and japanese. it surpasses existing event datasets in linguistic coverage and is the first event extraction dataset that includes burmese, indonesian, hebrew, somali, and nepali. we construct lemonade by extending expert annotations from acled. to ensure compatibility with nlp systems, we reannotated several event argument types, transformed event-centric annotations into a document-centric format, and generated descriptions for 10,707 entities to facilitate retrieval-based entity linking. each example in lemonade consists of a news article with its primary event annotated, following the document-level single-event configuration established in docee @xcite . the dataset includes 25 event types within the socio-political domain, ranging from peaceful protests to chemical weapons deployment. annotations comprise the event type and associated entity and non-entity arguments with their corresponding roles. appendix i details all event types and their arguments. ## zest: a novel abstractive entity linker zest employs a multi-stage approach to linking entities: first, it leverages information retrieval techniques to narrow down candidate entities; second, it filters these candidates based on their relevance; and finally, it assigns each entity to the appropriate event argument. for instance, in figure 1 , the entity "member of government of india" is assigned to the event argument victims. stage 1: entity retrieval. in the first stage, we construct a vector database by embedding all entities along with their descriptions using an embedding model. given an input document, zest utilizes the underlying llm to generate multiple queries for searching the entity database. these queries aim to closely match the descriptions of the gold entities, thus increasing the likelihood of retrieving relevant candidates. at test time, since the model does not have access to the gold entity descriptions, the llm approximates these descriptions based solely on the information available in the input document. for example, given the document shown in figure 1 , the system generates multiple queries for possible entities, including: "the former sarpanch of chaskhand panchayat, involved in a political rivalry with the current sarpanch, pratibha mallick," and "the state government of odisha, india, responsible for implementing development schemes and maintaining law and order in the region." the union of all entities retrieved by these queries is then passed to the next stage. stage 2: entity filtering. in the second stage, each candidate entity (along with its description) retrieved from stage 1 is evaluated using a dedicated prompt (see stage 3: entity assignment. in the final stage, the remaining entities are matched to their correct event arguments. to accomplish this, we employ another prompt (see ## conclusion in this paper, we introduced the task of abstractive event extraction (aee), a formulation that better aligns with the requirements of real-world event extraction applications. to support research in this direction, we created a large-scale, high-quality dataset for aee in 20 languages, derived from expert-annotated data provided by acled. our experiments demonstrate that existing spanbased models, such as gollie and onenet, are inherently unsuitable for the abstractive setting, consistently performing worse than models based on in-context learning. additionally, we proposed a novel zero-shot entity linking system, zest, which significantly narrows the performance gap in the abstractive entity linking (ael) subtask. despite this improvement, a substantial gap remains between zero-shot and fully supervised models. we hope that the release of lemonade will inspire further research, ultimately expanding the capabilities of future zeroshot event extraction models.