CLIO : Role-interactive Multi-event Head Attention Network for Document-level Event Extraction (2022)

Abstract: Transforming the large amounts of unstructured text on the Internet into structured event knowledge is a critical, yet unsolved goal of NLP, especially when addressing document-level text. Existing methods struggle in Document-level Event Extraction (DEE) due to its two intrinsic challenges: (a) Nested arguments, which means one argument is the sub-string of another one. (b) Multiple events, which indicates we should identify multiple events and assemble the arguments for them. In this paper, we propose a role-interactive multi-event head attention network (CLIO) to solve these two challenges jointly. The key idea is to map different events to multiple subspaces (i.e. multi-event head). In each event subspace, we draw the semantic representation of each role closer to its corresponding arguments, then we determine whether the current event exists. To further optimize event representation, we propose an event representation enhancing strategy to regularize pre-trained embedding space to be more isotropic. Our experiments on two widely used DEE datasets show that CLIO achieves consistent improvements over previous methods.

## introduction cognitive scientists believe that humans remember and understand reality primarily in terms of events @xcite . event studies are justifiably popular in natural language processing (nlp), such as event coreference resolution, event causality identification, and event extraction. event extraction is the process of extracting structured event knowledge from unstructured text and can be divided into sentence-level and document-level. sentence-level event extraction has demonstrated promising results in empirical evaluations. however, in real-world scenarios, a large number of event elements are expressed recently, researchers have shown an increased interest in dee. their works can be roughly divided into classification-based models @xcite @xcite , tagging-based models @xcite , and generation-based models @xcite @xcite . the state-of-the-art approach @xcite frames dee as a machine reading comprehension task, assisted by two data augmentation regimes. although scholars have made such valuable attempts in dee, current methods still struggle in dee due to the following crucial challenges: nested arguments: in a document, there are many nested arguments (i.e., one argument is the substring of another one) that belong to different roles. figure 1 gives an example. in the "trans-portation" event, "truck" (plays origin role) and "ryder truck" (plays vehicle role) are nested event arguments. according to our statistics, 14.23% and 13.94% of documents in the wikievents @xcite and rams @xcite datasets have nested arguments, respectively. unfortunately, these nested arguments can't be entirely identified by traditional tagging-based methods, which can not assign multiple labels to a token. multiple events: as shown in figure 1 , there are three kinds of events: "transportation", "ex-changebuysell", and "meet" in a single document, and dee should not only identify all events but also assign arguments to the corresponding events. the issue of multiple events is common in dee (86.88% of documents in the wikievents involve multiple events). what's more, the arguments of these events are uniformly scattered across sentences, making it hard to achieve accurate arguments assembling. previous works usually adopt a fixed document representation to detect all event types. however, different event types have different roles and arguments, and the emphasis of document representation should also be different. for the nested arguments, which usually belong to different roles, the intuition is that we should extract arguments for each role independently. assuming there are n roles in an event, we can perform n independent extractions by tagging arguments under each role. in this way, the argument substring "truck" of role "origin" and the argument "ryder truck" of role "vehicle" can be identified at the same time. to address the challenge of multiple events, an intuitive way is to independently detect each event type and assemble arguments for it. for one event type, argument extraction can be simpler due to the decrease in roles. on the contrary, using role information specific to this event type can better detect the current event type. we argue that these two challenges can be solved jointly by mapping each event type to a specific subspace. analogy to multi-head attention @xcite , we propose a role-interactive multi-event head attention network (clio) for dee. the most critical part in clio is role-interactive multi-event head attention module, which can solve the aforementioned two challenges jointly. first, our attention module works in a role-centric way. that is to say, for each role, we extract all of its corresponding arguments independently. in this way, a token can be assigned multiple role labels, which can perfectly solve nested arguments problem. second, our attention module assigns each event type a subspace by mapping it to each event head. in this way, we can independently detect each event type and assemble arguments for it, which can address the challenge of multiple events. in each event head, we use role information specific to this event to represent document. such event-specific document representation eases the difficulty of detecting multiple events from a single document. in summary, our contributions are as follows: ## methodology we first describe the task formalization of dee. formally, given an input document comprised of m words @xmath0, pre-defined event types @xmath1, and role categorizies @xmath2. the dee task aims to extract one or more event records: {event type : t, r 1 : [a 1 1 , a 2 1 , ...], ..., r i : [a 1 i , a 2 i , ...]}, where a 1 i is the first argument of role r i , and so on. figure 2 illustrates the architecture of clio, which consists of three key components: (1) roleinteractive multi-event head attention, (2) multiple events extraction, and (3) event representation enhancing. role-interactive multi-event head attention module can solve the challenges of nested arguments and multiple events simultaneously. ## experiments we evaluate our model's performance on the two commonly used dee benchmarks and compare to prior work. then we conduct an ablation study on how modules of our clio affect its performance on dee task. we also conduct case study to analyze qualitatively the advantages and disadvantages of our model. ## conclusion in this paper, we propose a role-interactive multievent head attention network (clio) for dee. by mapping different events to multiple subspaces, we decomposed dee into multiple substeps to handle nested arguments and multiple events. to further optimize event representation, we apply an event representation enhancing strategy to regularize pretrained embedding space to be more isotropic. experimental results show that clio can significantly outperform previous methods, especially when facing the specific challenges of dee. in future work, we would like to explore superior word representation specific to events.