GRIT : Generative Role-filler Transformers for Document-level Event Entity Extraction (2021)

Abstract: We revisit the classic problem of document-level role-filler entity extraction (REE) for template filling. We argue that sentence-level approaches are ill-suited to the task and introduce a generative transformer-based encoder-decoder framework (GRIT) that is designed to model context at the document level: it can make extraction decisions across sentence boundaries; is implicitly aware of noun phrase coreference structure, and has the capacity to respect cross-role dependencies in the template structure. We evaluate our approach on the MUC-4 dataset, and show that our model performs substantially better than prior work. We also show that our modeling choices contribute to model performance, e.g., by implicitly capturing linguistic knowledge such as recognizing coreferent entity mentions.

## introduction document-level template filling @xcite @xcite is a classic problem in information extraction (ie) and nlp @xcite . it is of great importance for automating many real-world tasks, such as event extraction from newswire @xcite . the complete task is generally tackled in two steps. the first step detects events in the article and assigns templates to each of them (template recognition); the second step performs role-filler entity extraction (ree) for filling in the templates. in this work we focus on the role-filler entity extraction (ree) sub-task of template filling (figure 1 ). 1 the input text describes a bombing event; the goal is to identify the entities that fill any of the roles associated with the event (e.g., the perpetrator, their organization, the weapon) by extracting a descriptive "mention" of it -a string from the document. in contrast to sentence-level event extraction (see, e.g., the ace evaluation @xcite ), document-level ree introduces several complications. first, role-filler entities must be extracted even if they never appear in the same sentence as an event trigger. in figure 1 , for example, the weapon and the first mention of the telephone company building (target) appear in a sentence that does not explicitly mention the explosion of the bomb. in addition, ree is ultimately an entity-based task -exactly one descriptive mention for each role-filler should be extracted even when the entity is referenced multiple times in connection with the event. the final output for the bombing example should, therefore, include just one of the "water pipes" references, and one of the three alternative descriptions of the perpind and the second target, the telephone company building. as a result of these complications, end-to-end sentence-level event extraction models @xcite , which dominate the literature, are ill-suited for the ree task, which calls for models that encode information and track entities across a longer context. fortunately, neural models for event extraction that have the ability to model longer contexts have been developed. @xcite , for example, extend standard contextualized representations @xcite to produce a documentlevel sequence tagging model for event argument extraction. both approaches show improvements in performance over sentence-level models on event extraction. regrettably, these approaches (as well as most sentence-level methods) handle each candidate role-filler prediction in isolation. consequently, they cannot easily model the coreference structure required to limit spurious role-filler mention extractions. nor can they easily exploit semantic dependencies between closely related roles like the perpind and the perporg, which can share a portion of the same entity span. "shining path members", for instance, describes the perpind in figure 1 , and its sub-phrase, "shining path", describes the associated perporg. contributions in this work we revisit the classic but recently under-studied problem of documentlevel role-filler entity extraction problem and introduce a novel end-to-end generative transformer model -the "generative role-filler transformer" (grit) (figure 2 ). empirically, our model outperforms substantially strong baseline models. we also demonstrate that grit is better than existing document-level event extraction approaches at capturing linguistic properties critical for the task, including coreference between entity mentions and cross-role extraction dependencies. foot_2 2 related work sentence-level event extraction most work in event extraction has focused on the ace sentencelevel event task @xcite , which requires the detection of an event trigger and extraction of its arguments from within a single sentence. previous state-of-the-art methods include @xcite and @xcite , which explored a variety of hand-designed features. more recently, neural network based models such as recurrent neural networks @xcite , convolutional neural networks (nguyen and grishman, 2015; @xcite and attention mechanisms @xcite have also been shown to help improve performance. beyond the taskspecific features learned by the deep neural models, @xcite and @xcite also utilize pre-trained contextualized representations. only a few models have gone beyond individual sentences to make decisions. @xcite and @xcite utilize event type co-occurrence patterns to propagate event classification decisions. @xcite propose to learn within-event (sentence) structures for jointly extracting events and entities within a document context. similarly, from a methodological perspective, our grit model also learns structured information, but it learns the dependencies between role-filler entity mentions and between different roles. @xcite and @xcite leverage document embeddings as additional features to aid event detection. although the approaches above make decisions with cross-sentence information, their extractions are still done the sentence level. document-level ie document-level event rolefiller mention extraction has been explored in recent work, using hand-designed features for both local and additional context @xcite @xcite , and with end-to-end sequence tagging based models with contextualized pre-trained representations @xcite . these efforts are the most related to our work. the key difference is that our work focuses on a more challenging, and more realistic, setting: extracting role-filler entities rather than lists of role-filler mentions that are not grouped according to their associated entity. also on a related note, chambers and jurafsky (2011), chambers (2013 et al. (2020) introduce rams dataset for multi-sentence argument mention linking, while we focus on entity-level extraction in our work. different from work on joint modeling @xcite and multi-task learning @xcite setting for extracting entities and relations, through the generative modeling setup, our grit model implicitly captures (non-)coreference relations between noun phrases, without relying on the cross-sentence coreference and relation annotations during training. neural generative models with a shared module for encoder and decoder our grit model uses one shared transformer module for both the encoder and decoder, which is simple and effective. for the machine translation task, @xcite propose a model which shares the parameters of each layer between the encoder and decoder to regularize and coordinate the learning. @xcite presents a new unified pre-trained language model that can be fine-tuned for both nlu and nlg tasks. similar to our work, they also introduce different masking strategies for different kinds of tasks (see section5). ## the role-filler entity extraction task and evaluation metric we base the ree task on the original muc foot_3 formulation @xcite , but simplify it as done in prior research @xcite . in particular, we assume that one generic template should be produced for each document: for documents that recount more than one event, the extracted role-filler entities for each are merged into a single event template. second, we focus on entity-based roles with string-based fillers foot_4 . there are five such roles for muc-4: perpetrator individuals (perpind), perpetrator organizations (perporg), physical targets (target), victims (victim) and weapons (weapon). these event roles represent the agents, patients, and instruments associated with terrorism events @xcite . ## ree as sequence generation we treat document-level ree as a sequence-tosequence task @xcite in order to better model the cross-role dependencies and crosssentence noun phrase coreference structure. we first transform the task definition into a source and target sequence. as shown in figure 2 , the source sequence simply consists of the tokens of the original document prepended with a "classification" token (i.e., @xcite in bert), and appended with a separator token (i.e., [sep] in bert). the target sequence is the concatenation of target extractions for each role, separated by the separator token. for each role, the target extraction consists of the first mention's ## model: generative role-filler transformer (grit) our model is shown in figure 2 . it consists of two parts: the encoder (left) for the source tokens; and the decoder (right) for the target tokens. instead of using a sequence-to-sequence learning architecture with separate modules @xcite , we use a single pretrained transformer model @xcite for both parts, and introduce no additional fine-tuned parameters. pointer embeddings the first change to the model is to ensure that the decoder is aware of where its previous predictions come from in the source document, an approach we call "pointer embeddings". similar to bert, the input to the model consists of the sum of token, position and segment embeddings. however, for the position we use the corresponding source token's position. for example, for the word "two", the target tokens would have the identical position embedding of the word "two" in the source document. interestingly, we do not use any explicit target position embeddings, but instead separate each role with a [sep] token. empirically, we find that the model is able to use these separators to learn which role to fill and which mentions have filled previous roles. our encoder's embedding layer uses standard bert embedding layer, which applied to the source document tokens. to denote boundary between source and target tokens, we use sequence a (first sequence) segment embeddings for the source tokens, we use sequence b (second sequence) segment embeddings for the target tokens. bert target tokens + pointer embeddings model [cls] … a bomb exploded … destroying some [water pipes]. … the bomb … was placed in the back of the [pilmai [telephone company building]]. … the explosion caused damages to the [telephone company offices]. it also destroyed a [public telephone booth] and [water pipes] … the bomb was planted by [[two men] wearing sports clothes], escaped. … later identified as [[shining path] members]… [sep] pointer selection source tokens two men [sep] shining path [sep] water pipes pilmai building @xmath0... target t o k e n s … tokens … asks: figure 3: partially causal masking strategy (m). (white cell: unmasked; grey cell: masked). we pass the source document tokens through the encoder's embedding layer, to obtain their embeddings x 0 , x 1 , ..., x m . we pass the target tokens y 0 , y 1 , ..., y n through the decoder's embedding layer, to obtain y 0 , y 1 , ..., y n . ## experimental setup we conduct evaluations on the muc-4 dataset (1992), and compare to recent competitive end-to-end models @xcite in ie (section 7). besides the normal evaluation, we are also interested in how well our grit model captures coreference linguistic knowledge, and comparison with the prior models. in section 8, we present relevant evaluations on the subset of test documents. ## results in ## discussion how well do the models capture coreference relations between mentions? we also conduct targeted evaluations on subsets of test documents whose gold extractions come with coreferent mentions. from left to right in ## discussion how well do the models capture coreference relations between mentions we also see our model recognizes the coreference relation between candidate role-filler entities, while the baselines don't. this demonstrates that our model is better at capturing the (non)-coreference relation between role-filler entities. it also proves the advantage of generative modeling (over modeling one candidate role-filler entity's role in isolation). our model correctly extracts the two role-filler entities for perporg: "farc" and "popular liberation army", which are closely related to the per-pind entity "guerrilla". while the dygie++ and nst both miss the entities for perporg. ## conclusion we revisit the classic and challenging problem of document-level role-filler entity extraction (ree), and find that there is still room for improvement. we introduce an effective end-to-end transformer based generative model, which learns the document representation and encodes the dependency between role-filler entities and between event roles. it outperforms the baselines on the task and better captures the coreference linguistic phenomena. in the future, it would be interesting to investigate how to enable the model to also do template recognition.