DORE : Document Ordered Relation Extraction based on Generative Framework (2022)

Abstract: In recent years, there is a surge of generation-based information extraction work, which allows a more direct use of pre-trained language models and efficiently captures output dependencies. However, previous generative methods using lexical representation do not naturally fit document-level relation extraction (DocRE) where there are multiple entities and relational facts. In this paper, we investigate the root cause of the underwhelming performance of the existing generative DocRE models and discover that the culprit is the inadequacy of the training paradigm, instead of the capacities of the models. We propose to generate a symbolic and ordered sequence from the relation matrix which is deterministic and easier for model to learn. Moreover, we design a parallel row generation method to process overlong target sequences. Besides, we introduce several negative sampling strategies to improve the performance with balanced signals. Experimental results on four datasets show that our proposed method can improve the performance of the generative DocRE models.

## introduction document-level relation extraction (docre) is a fundamental information extraction (ie) task which aims to extract relational facts among entities across multiple sentences. for ie, most previous approaches are classification-based, which first extract features of certain objects using pretrained language models and then classify according to the merged features. recent years have witnessed a rising trend of regarding the task of ie as a sequence generation problem, linearizing the extracted structures as a sequence. compared to classification-based methods, generative framework extracts features and classifies simultaneously, allowing a more direct use of latent knowledge in pre-trained language models without an untrained classification module on the top. besides, the generation process can naturally recover high-order dependencies when generating the output step by step. generation-based methods have been successfully adapted to many settings including universal ie which intends to solve several ie tasks in a unified way @xcite , lowresource @xcite , and transfer learning @xcite , and have achieved competitive results on most sentence-level benchmarks @xcite and documentlevel event extraction task @xcite . considering that generative framework is simple and effective, prior work adopts it for docre @xcite . this line of work features lexical generation since they use natural language to represent entities and relations, which directly borrows from text generation tasks @xcite . also, they need special separator tokens to distinguish token spans. however, the lexical generation paradigm dose not perfectly fit the more complex docre task, where the source sequence contains numerous entities and relations (e.g., a document can contain up to about 40 relation instances on docred), leading to a performance gap between generation-based and classification-based methods. our experiment verifies that the generative baseline performs 6.00 points worse than classification-based methods on docred dataset @xcite . the lexical generation paradigm faces two significant challenges, which impede its performance. to alleviate issues in the lexical generation paradigm, we treat generative docre as deterministically generating a relation matrix where each cell corresponds to an entity pair with predefined relation or no relation. the paradigm, which we call dore (document ordered relation extraction), assigns each entity and relation a special id and linearizes the relation matrix in the row-column order, resulting a symbolic ordered sequence. it is much easier to learn and control generation. in addition, the paradigm is able to resolve overlong output sequences in a concise way when generating rows of the relation matrix in parallel. besides, we show that the loss function taken from previous work is imbalanced for the complicated docre, and we introduce several negative sampling strategies to mitigate it. taken together, we find that the underwhelming performance of generative framework for docre comes from the improper training and generation ways instead of the model architecture. we conduct experiments on four popular docre benchmarks. we improve the generative model's f 1 score from 51.36 to 60.67 for docred by changing training paradigm only, and further improve it to 65.26 with distantly supervised training data (officially collected by docred). besides, we bridge the performance gap between generationbased and classification-based methods on cdr @xcite and gda @xcite by obtaining 72.6 and 85.3 f 1 score, individually. we also achieve new state-of-the-art results on scirex @xcite both with gold inputs and end-toend for binary and 4-ary relation extraction. our work brings generative framework to docre into a performance region that matches classificationbased approaches, with the added advantage of supporting high-order relation discovery afforded by the nature of sequence-to-sequence models. ## conclusion we propose a new generative paradigm dore for docre. dore adopts a symbolic and ordered sequence representation, establishing a clean connection between the sequence generation and docre. we also introduce parallel row generation and several negative sampling methods to improve the effectiveness and efficiency. experiments on four docre datasets demonstrate that our method can substantially improve generative models without changing their designs.