Reading the Manual: Event Extraction as Definition Comprehension (2020)

Abstract: We ask whether text understanding has progressed to where we may extract event information through incremental refinement of bleached statements derived from annotation manuals. Such a capability would allow for the trivial construction and extension of an extraction framework by intended end-users through declarations such as, “Some person was born in some location at some time.” We introduce an example of a model that employs such statements, with experiments illustrating we can extract events under closed ontologies and generalize to unseen event types simply by reading new definitions.

## introduction this work is aimed at the disconnect between how human annotators and machines carry out information extraction: humans read annotation manuals consisting of guidelines and illustrative examples then label data, whereas machines label data (by making predictions) based purely on previously seen examples (figure 1 ). we explore the feasibility of building a model that has access to information derived from annotation manuals. specifically we focus on the task of event extraction and convert annotation guidelines describing event types into natural language bleached statements. an example bleached statement for the ace 2005 @xcite life:be-born event type is: we are motivated to consider these statements owing to the rapid progress in sentence-level representation learning and machine reading comprehension: can a contemporary encoder understand event statements well enough that their derived representation may be directly employed in extraction? bleached statements are straightforward to write and accommodate various levels of expressiveness in both the choice of arguments present and in the lexicalization of the trigger. foot_0 these features allow for easy adaptation as an ontology changes: simply introduce new or modified statements. in the case where the amount of labeled examples is small or non-existent, a bleached statement serves as a sort r m f x c b j 8 c e 6 d d i b q m 7 w 9 2 m h t q p 6 j / q v + u / 5 n e r s 2 m y t 5 e s y s + t 9 / 9 p o c t @xmath0 r m f x c b j 8 c e 6 d d i b q m 7 w 9 2 m h t q p 6 j / q v + u / 5 n e r s 2 m y t 5 e s y s + t 9 / 9 p o c t @xmath1 r m f x c b j 8 c e 6 d d i b q m 7 w 9 2 m h t q p 6 j / q v + u / 5 n e r s 2 m y t 5 e s y s + t 9 / 9 p o c t @xmath2 unrest continued in other parts of the country. americans sought to bring calm to mosul, where u.s. troops killed 17 people in clashes earlier in the week. brooks, briefing reporters at the u.s. central command headquarters, also disclosed that marines had broken up violent clashes on wednesday in tikrit, saddam's hometown. text statement round 1 round 2 round 3 round 4 round 5 life:die 2 6 6 6 6 6 6 6 6 6 4 trigger : "killed" agent : "u.s. troops" victim : "17 people" place : "mosul" time : "earlier in the week" < l a t e x i t s h a 1 _ b a s e 6 4 = " the bleached statement is incrementally populated with values from the text (in the order denoted by the superscripts), and not all event arguments are supported by the text. the grayed out text in the paragraph is given for context to the reader, but our model operates on single-sentence contexts. as an example of such a solution, we propose a model which incrementally populates bleached statements by querying partially filled statements against text. this strategy is similar to the tasks of machine reading comprehension (mrc) and question answering (qa), in which an answer span is predicted in response to a question about a document. conceptually this may also be considered a form of incremental recognizing textual entailment (rte) @xcite where we iteratively refine a hypothesis that is supported by the document context. experimental results demonstrate that zero-and few-shot event extraction are feasible with this approach. while our intent here is exploratory, we manage to achieve state-of-the-art performance on trigger identification and trigger classification on the ace 2005 dataset. the contributions of this work are: (i) a novel approach to event extraction that takes into account annotation guidelines through bleached statements; (ii) a multiple-span selection model that demonstrates the feasibility of the approach for event extraction as well as for zero-and few-shot settings. ## background event extraction is traditionally viewed as three subtasks: (1) event trigger detection, where triggers of events (words that most clearly express the occurrences of events) are detected; (2) entity men-tion detection, where all potential arguments (entity mentions) to events are detected; and (3) argument role prediction, where relations between detected arguments and trigger words are recognized with respect to each event type's defined set of roles. much prior work adopts a pipelined approach to these 3 subtasks or focuses on a subset of the subtasks based on gold entity mention spans. these include feature-based approaches @xcite @xcite @xcite and neural approaches @xcite @xcite @xcite . because pipelined approaches suffer from error propagation in which the error from earlier subtasks (e.g. entity mention detection) is inherited by later subtasks, joint modeling of the 3 subtasks has been attempted. @xcite attempts to jointly model the three components with hand-crafted features, but still need to detect entity mentions and event triggers separately. nguyen and nguyen (2019) jointly models the three tasks using neural networks with shared underlying representations. the models proposed in these two works are the baselines used in this paper. @xcite approach zero-shot event extraction by stipulating a graph structure for each event type and finding the event type graph structure whose learned representation most closely matches the learned representation of the parsed amr @xcite structure of a text. in contrast, our approach forgoes explicit graphstructured semantic representations such as amr. researchers have introduced large question an-swering (qa) / machine reading comprehension (mrc) datasets in a cloze style @xcite , where a query sentence contains a placeholder and the model fills the blank. our work can be viewed as an extension to such work, where multiple placeholders are extracted. @xcite casts relation extraction as multiturn qa with natural language questions, where in each turn one argument of the relation is found.. the method requires writing a question for each entity type and each relation type. in @xcite , sets of crowdsourced paraphrastic questions are written for each relation type in the ontology. in contrast, for each event type we use a single declarative bleached statement derived from the annotation guidelines. @xcite proposes a model for relation extraction by filling in two blanks given a contextual relation statement. these three methods focus on binary relation extraction, and do not readily generalize to -ary events or relations. our approach naturally supports variable arity events and relations. ## problem formulation the event extraction task as defined in the ace 2005 dataset also requires finding an event triggera span in the text that most clearly expresses the event's occurrence. in this example, the trigger is the word "killed". for a consistent implementation, we consider the trigger to be a special argument of the event, with role name trigger. formally, the task is: given a bleached statement , its placeholder dictionary , and text tokens , return a dictionary ˆ that contains the event trigger and the the extracted arguments. such a result is shown in the bottom right of figure 2 . note that the instrument role is not filled in the example because the model does not find a span to fill it. ## approach given a bleached statement with multiple placeholders, we do not fill the placeholders in parallelinstead, we fill them incrementally in an enforced order. 4 in each step, the model attempts to fill a single focused placeholder, which is replaced by the extracted span(s) thereby creating a refined statement (see figure 2 ). in this work we fill the placeholders in the statement from left to right and leave other orders as future work. formally, in each round, our model returns multiple arguments (see "multiple argument selector" section below) for each placeholder: if is the empty set, then the model did not find an appropriate text span to replace the placeholder. if the answer set is not empty, we replace the placeholder with the extracted span. note that in some cases, there can be more than one argument that fits a role. consider the following bleached statement (for the ace 2005 event life:marry), focused on the first placeholder "some people": kim and pat person 1 married in some location place 2 at some time time 3 we run this iterative process until all roles of an event are visited. an advantage of this method is that during the incremental refinement process, the statement always remains a natural language sentence. the incremental process for extracting event arguments is formalized in algorithm 1, given the initial bleached statement , the role dictionary , and the text to extract from. annotation manuals of interest usually define multiple event types. for each event type described in the manual, we require a bleached statement and a role dictionary . these together form our ontology @xmath3, , )}. to perform full event extraction (algorithm 2), we first run a trigger detection model for all event types (see algorithm 2 event extraction input: ontology o, text output: extracted event structures e function extractevents(o, ) e ← for ( , , ) ∈ o do triggers ← triggerid( , , ) @xmath4, ) ← extractargs( , , ) ˆ ← ∪ (trigger : ) e ← e ∪ ˆ end for end for return e ## conclusion & future work we present an approach to event extraction that uses bleached statements to give a model access to information contained in annotation manuals. our model incrementally refines the statements with values extracted from text. we also demonstrate the feasibility of making predictions on event types seen rarely or not at all. future work can apply our approach to -ary relation extraction.