Document-level Entity-based Extraction as Template Generation (2021)

Abstract: Document-level entity-based extraction (EE), aiming at extracting entity-centric information such as entity roles and entity relations, is key to automatic knowledge acquisition from text corpora for various domains. Most document-level EE systems build extractive models, which struggle to model long-term dependencies among entities at the document level. To address this issue, we propose a generative framework for two document-level EE tasks: role-filler entity extraction (REE) and relation extraction (RE). We first formulate them as a template generation problem, allowing models to efficiently capture cross-entity dependencies, exploit label semantics, and avoid the exponential computation complexity of identifying N-ary relations. A novel cross-attention guided copy mechanism, TopK Copy, is incorporated into a pre-trained sequence-to-sequence model to enhance the capabilities of identifying key information in the input document. Experiments done on the MUC-4 and SciREX dataset show new state-of-the-art results on REE (+3.26%), binary RE (+4.8%), and 4-ary RE (+2.7%) in F1 score.

## introduction document-level entity-based extraction (ee) are tasks that extract entity-centric information, such as entities and their relations, from unstructured text across multiple sentences. with the rise of big data in recent years, document-level ee is growing in importance with applications such as understanding clinical reports @xcite , extracting document-level events @xcite , and building knowledge graphs from journals @xcite . in this work, we focus on two classic tasks of document-level ee: role-filler entity extraction (ree) and relation extraction @xcite . recent works on document-level ee usually build 1 the source code is publicly available at https:// github.com/pluslabnlp/tempgen @xcite , on a relation extraction example from scirex. the task is to extract entities and identify which entities are related from the given scientific article. due to the long distances between entities, scirex-p struggles to extract the right entity pair that has a relation, while our approach correctly identifies them. this reflects our method's advantage in modeling long-term cross-entity dependencies. motivated by these challenges, we propose to formulate ree and re tasks as template generation. due to the autoregressive nature of generative setup, this formulation makes dependencies among the output entities easier to capture compared to sequence tagging methods. moreover, label names are incorporated into the decoder targets for exploiting label semantics not present in the extractive counterparts. furthermore, for tasks that involve the identification of n -ary relations, this formulation significantly alleviates the computational complexity of comparing exponential combinations of entities. a generative framework, cross-attention guided template generation @xcite , that incorporates a novel copy mechanism into a pretrained sequence-to-sequence model is proposed to solve the template generation problem effectively. our contributions can be summarized as follows: ## tasks this section gives an overview of the two document-level ee tasks we tackled in this work: role-filler entity extraction (ree) and relation extraction (re). ## proposed methods in this section, we first illustrate how the ree and re tasks can be framed as a template generation problem. this formulation then allows us to capture cross-entity dependencies easily with our proposed generative model, a pre-trained sequence-tosequence model integrated with a copy mechanism. ## inference time comparison as discussed earlier, tempgen can significantly reduce the exponential computational complexity of document-level n-ary relation identification. to illustrate this, we compared the inference time between tempgen and two other systems, tanl and scirex-p, on the scirex 4-ary re task. as shown in figure 7 , tempgen drastically shortens the inference time by around 39 times compared to scirex-p. tanl also runs much faster than scirex-p, but is still around 4 times slower than tempgen. this is resulted from the fact that tanl generates the entire input document in addition to entity and relation labels, which is much longer than tempgen's generated sequences. ## number of parameters figure 8 shows the number of parameters of different models. grit, with the same size of bert-base, has the least number of parameters among all models. dygie++ and scirex-p have slightly more parameters than grit due to the additional linear layers for constructing classifiers. the two generative models, tanl and tempgen, have the most parameters, thanks to the larger vocab size (30522 → 50265), larger positional embedding matrix (512 → 1024), and cross-attention modules in bart-base. ## related works in the following sections, we will first discuss a few important works on the ree task and document-level re task. then, we will dive into a few works that uses a similar sequence generative approach to various document-level ie tasks. ## conclusion we have proposed tempgen, a framework that frames document-level ree and re tasks as a template generation task. a copy mechanism that takes the top-k important cross-attentions as copy distributions is incorporated into bart for capturing key information in the input document. experimental results on muc-4 and scirex showed that tempgen outperforms prior approaches on rolefiller entity extraction and end-to-end documentlevel relation extraction tasks. under different amount of training data, tempgen demonstrates robustness across all settings, while being advantageous in lower-resource regime.