Few-Shot Document-Level Relation Extraction (2022)

Abstract: We present FREDo, a few-shot document-level relation extraction (FSDLRE) benchmark. As opposed to existing benchmarks which are built on sentence-level relation extraction corpora, we argue that document-level corpora provide more realism, particularly regarding none-of-the-above (NOTA) distributions. Therefore, we propose a set of FSDLRE tasks and construct a benchmark based on two existing supervised learning data sets, DocRED and sciERC. We adapt the state-of-the-art sentence-level method MNAV to the document-level and develop it further for improved domain adaptation. We find FSDLRE to be a challenging setting with interesting new characteristics such as the ability to sample NOTA instances from the support set. The data, code, and trained models are available online ( https://github.com/nicpopovic/FREDo ).

## introduction the goal of relation extraction is to detect and classify relations between entities in a text according to a predefined schema. the schema, defining which relation types are relevant is highly dependent on the specific application and domain. supervised learning methods for relation extraction @xcite @xcite @xcite , which have advanced rapidly since the introduction of pretrained language models such as bert @xcite , need large corpora of annotated relation instances to learn a schema. since annotating data sets for relation extraction manually is expensive and time consuming, few-shot learning for relation extraction represents a promising solution for relation extraction at scale. while the general n -way k-shot few-shot learning framework is relatively well defined and appears easy to apply to relation extraction, constructing realistic benchmark tasks has proven to be challenging. one of the core difficulties of establishing a realistic benchmark task for few-shot relation extraction is correctly modelling the most frequent situation a relation extraction system will encounter, none-of-the-above (nota) detection. nota refers to the case in which a candidate pair of entities does not hold any of the relations defined in the schema, a situation which is far more common than its reverse (for the document-level data set docred @xcite , 96.84% of candidate entity pairs are nota cases). while initial benchmarks @xcite ignored this scenario altogether, researchers working on few-shot relation extraction have pushed for more realistic nota modeling in tasks and developed methods that can better detect nota instances @xcite . parallel to the outlined efforts towards realistic few-shot relation extraction benchmarks, research into supervised relation extraction has moved from sentence-level tasks, relation extraction within single sentences, to document-level relation extraction. the push towards document-level relation extraction is motivated by (1) extracting more complex, cross-sentence relations and (2) information extraction at scale. the latter is driven by an inherent challenge when increasing the scope from single sentences to multiple sentences: the number of entities involved increases and with that comes a quadratic increase in candidate entity pairs. while sentence-level approaches typically evaluate each candidate entity pair individually, this strategy is infeasible at the document-level (docred contains an average of 393.5 candidate entity pairs per document, compared to only 2 for many sentence leveltasks). in addition to the increased computation requirements, this results in a drastic increase in the amount of nota examples in a given query, demanding new methods of handling the imbalances that come with this change of distribution @xcite . all current few-shot relation extraction bench-marks are based on sentence-level tasks. we argue that moving few-shot relation extraction from the sentence-level to the document-level: (1) brings with it as an inherent characteristic the more realistic nota distribution which prior work has looked to emulate and (2) will make the resulting methods more suitable for large scale information extraction. in this work, we therefore define a new set of few-shot learning tasks for document-level relation extraction and design a strategy for creating realistic benchmarks from annotated document corpora. applying the above to the data sets docred @xcite and scierc @xcite , we construct a few-shot document-level relation extraction (fsdlre) benchmark, fredo, consisting of two main tasks, an in-domain and a crossdomain task requiring domain adaptation. finally, building on the state-of-the-art few-shot relation extraction approach mnav @xcite and document-level relation extraction concepts @xcite , we develop two approaches for tackling the above tasks. we begin by outlining key related work in section 2. in section 3 we give a description of the proposed tasks. next, in section 4 we explain the construction of the benchmark, fredo, followed by an overview of the proposed methods (section 5), an analysis and discussion of the observed results (section 6), and the conclusion (section 7). ## related work to the best of our knowledge, all current few-shot relation extraction benchmarks @xcite @xcite focus on extracting relations from single sentences. fewrel @xcite introduces a relation extraction benchmark in the n -way k-shot setting @xcite in which a relation instance is to be assigned to one of n classes given only k examples for each of the classes. in this setting human performance was quickly surpassed @xcite , leading gao et al. @xcite to create fewrel 2.0 in an effort to increase the difficulty by adding a domain adaptation task, as well as a nota detection task. sabo et al. @xcite argue that the way fewrel 2.0 models nota cases is not realistic due to the way nota instances are sampled, develop a framework for creating more realistic benchmarks and propose building such a benchmark using the sentence-level data set tacred @xcite . tran et al. @xcite forego labeled training data altogether and focus on the one-shot and weakly-supervised classification setting without nota cases. while multiple relation extraction data sets based on annotated documents, rather than single sentences, are available in the form of cdr @xcite , scierc @xcite , scirex @xcite , dialogre @xcite , and gda @xcite , the introduction of the large scale data set docred @xcite seems to have significantly increased research interest into supervised relation extraction at the document-level more recently @xcite @xcite . since documents contain considerably more entities than individual sentences and the amount of candidate entity pairs increases quadratically with the amount of entities, applying sentencelevel methods to document-level tasks is not feasible. document-level relation extraction approaches therefore use a different architecture @xcite than sentence-level approaches. another challenge is the large imbalance in the amount of positive and negative examples of relations encountered during training. some researchers approach the problem by resampling training examples to counteract the imbalance @xcite , while others use more specialized solutions, such as modified loss functions @xcite . ## task description in document-level relation extraction the task is to return a set s containing all valid triples of the format (e h , r i , e t ) for a document d. here, e h and e t are the head-and tail-entity of a relation instance, respectively, and r @xmath0, with r being the set of relation types for which instances are to be extracted. the positions of any entity mentions, as well as any co-reference clusters are provided as part of the input foot_1 . in both supervised learning and few-shot learning the documents used at test time are sampled from a different corpus than those used at training time. the added complexity in few-shot learning is caused (1) by a change in the set of relation types r between training and test time, and (2) by a much smaller amount of annotated examples given for each relation type. ## conclusion in order to encourage the development of few-shot relation extraction approaches which are useful in real world scenarios, we propose fredo, a few-shot document-level relation extraction benchmark. by moving to the document-level, the settings become more realistic, a problem which existing benchmarks are struggling with. for both in-domain and cross-domain tasks we present an approach which performs better than a simple baseline. our experiments confirm that, even though some existing benchmarks imply that impressive, even superhuman performance can already be achieved in few-shot relation extraction, realistic tasks are very difficult using current approaches and that significant advances are required for fewshot relation extraction approaches to be usable in a real word scenario. in providing a benchmark that reveals this performance gap, we look to pave the way towards new methods with a potentially high impact on domain-specific and cross-domain relation extraction at scale.