Template Filling with Generative Transformers (2021)

Abstract: Template filling is generally tackled by a pipeline of two separate supervised systems – one for role-filler extraction and another for template/event recognition. Since pipelines consider events in isolation, they can suffer from error propagation. We introduce a framework based on end-to-end generative transformers for this task (i.e., GTT). It naturally models the dependence between entities both within a single event and across the multiple events described in a document. Experiments demonstrate that this framework substantially outperforms pipeline-based approaches, and other neural end-to-end baselines that do not model between-event dependencies. We further show that our framework specifically improves performance on documents containing multiple events.

## introduction the classic template-filling task in information extraction involves extracting event-based templates from documents @xcite @xcite . it is usually tackled by a pipeline of two separate systems, one for role-filler entity extraction -extracting event-relevant entities (e.g., noun phrases) from the document; another for template/event recognition -assigning each of the candidate role-fillers to the event(s)/template(s) that it participates in and identifying the type of each event/template. simplifications of the task @xcite @xcite assume that there is one generic template and focus only on role-filler entity extraction. however, real documents often describe multiple events (figure 1 ). from the example, we can observe that between-event dependencies are important (e.g., a single organization can participate in multiple events) and can span the entire document (e.g., event-specific targets can be distant from their several attacks were carried out in la paz last night, one in front of government house ... the self-styled "zarate armed forces" sent simultaneous written messages to the media, calling on the people to oppose ... the first attack occurred at 22:30 in front of the economic ministry, just before president paz zamora concluded his message to ... roberto barbery, has reported that dynamite sticks were hurled from a car. the second attack occurred at 23:35, just after the cabinet members had left government house where they had listened to the presidential message. a bomb was placed outside government house in the parking lot that is used by cabinet ministers. the police ... as of 5:00 today, people found that an old shack on the estate was set ablaze, event 2 template bombing perpetrator indiv. -perpetrator org zarate armed forces physical target government house weapon bomb victim -event 1 template attack perpetrator indiv. -perpetrator org zarate armed forces physical target economic ministry weapon dynamite sticks victim -event 3 template arson perpetrator indiv. -perpetrator org zarate armed forces physical target old shack weapon -victim -figure 1: the template-filling task. role-filler entity extraction is shown on the left, and template recognition is shown on the right. our system performs both of these document-level tasks with a single end-to-end model. to naturally model between-event dependencies across a document for template filling, we propose a framework called "gtt" based on generative transformers (figure 2 ). to our best knowledge, this is the first attempt to build an end-to-end learning framework for this task. we build our framework upon grit @xcite , which tackles role-filler entity extraction (ree), but not template/event recognition. grit performs ree by "generating" a sequence of role-filler entities, one role at a time in a prescribed manner. for the template-filling setting, we first extend the grit approach to include tokens representing event types ## methodology our framework is illustrated in figure 2 . first we transform the template filling task into a se-quence generation problem. then, we train the base model on the source-target sequence pairs, and apply the model to generate the sequence; finally the sequence is transformed back to structured templates. ## experiments we conduct evaluations on the muc-4 dataset (1992). muc-4 consists of 1,700 documents with associated templates. we follow prior work in split: 1,300 documents for training, 200 documents (tst1+tst2) as the development set and 200 documents (tst3+tst4) as the test set. we use the metric for template filling @xcite and, as in previous work, map predicted templates to gold templates during evaluation so as to optimize scores. we follow content-based mapping restrictions, i.e., the event type of the template is considered essential for the mapping to occur. 1 missing template's slots are scored as missing, spurious template's slots are scored as spurious. note that in our work, since we do not extract the set fillers other than the event/template type, they do not affect the performance. baselines and additional related work as an ablation baseline, we employ a pipeline, grit-pipeline, that first uses the grit model for rolefiller entity extraction, and then assigns event types to each of the entities as a multi-label classification problem. we assign types by transforming the problem to multi-class classification (mcc) @xcite . as there are 6 event types (i.e., kidnapping, attack, bombing, robbery, arson, forced work stoppage) in muc-4, we use 2 6 labels for the mcc problem. ## results and analysis results on the full test set are shown in per-slot f1 score is reported in ## conclusion we revisit the classic nlp problem of template filling and propose an end-to-end learning framework called gtt. through modeling events relation, our approach better captures dependencies across the document and performs substantially better on multi-event documents.