Towards Realistic Few-Shot Relation Extraction (2021)

Abstract: In recent years, few-shot models have been applied successfully to a variety of NLP tasks. Han et al. (2018) introduced a few-shot learning framework for relation classification, and since then, several models have surpassed human performance on this task, leading to the impression that few-shot relation classification is solved. In this paper we take a deeper look at the efficacy of strong few-shot classification models in the more common relation extraction setting, and show that typical few-shot evaluation metrics obscure a wide variability in performance across relations. In particular, we find that state of the art few-shot relation classification models overly rely on entity type information, and propose modifications to the training routine to encourage models to better discriminate between relations involving similar entity types.

## introduction few-shot approaches have been explored in a variety of natural language processing (nlp) tasks, such as machine translation @xcite and textual entailment @xcite , as well as an assortment of classification and natural language inference tasks @xcite . the introduction of large, pretrained transformer language models in nlp increased the promise of building systems that can perform complex nlp tasks from only a small number of training examples @xcite . @xcite introduced a few-shot learning framework for relation classification @xcite , and recently several systems have achieved near-human performance on this task -in some settings even exceeding it. 1 these stunning results might give the impression that fewrel has been solved, and that these systems can be used to extract any relation of interest from a collection of text (e.g., to populate a knowledge-base from web documents) with only a few example instances. in this paper, we take a deeper look at the applicability of high performing few-shot relation classification models in a more realistic relation extraction (re) setting. we find that although transformer-based models achieve high accuracy at the fewrel task, this obscures a wide variability in performance across relation types. in particular, we find that state-of-the-art fewrel models heavily rely on entity type information, and thus are unable to discriminate between many types of relations that are trivial for humans (e.g., spouse-of vs. childof). however, we find that enriching the training data with relations with similar entity types forces the model to attend less to entity type information, and in a ranking evaluation, improves performance on unseen relations by up to 24% precision at 50, absolute. ## related work several recent papers have analyzed the weaknesses of sota supervised relation extraction systems, primarily on the tacred dataset @xcite . rosenman et al. (2020) list several "lazy" strategies employed by supervised sota models in the tacred challenge, including the "entity-type heuristic" which relies solely on entity types, ignoring context. @xcite perform in-depth error analysis and show that many errors stem from confusing relations with identical (coarse) type signatures, and ignoring context. tran et al. (2020) present a system that uses only entity information (in the form of unsupervised cluster ids) to match sota results on tacred. to the best of our knowledge, there has not been a similar error analysis for few-shot classification models. ## conclusions in this work we explored the applicability of fewshot relation classification models in a relation extraction setting. we showed that high classification accuracy does not translate to high extraction performance, due to the reliance of few-shot models on entity type information. as a result, the models tend to perform poorly on relations involving broad entity types, such as people, locations, or dates. by explicitly adding confusable relations at training time, we force the model to rely less heavily on entity types, and consequently discriminate between relations with similar argument types. further modifications to the training sampler that encourage the model to downweight entity type information are the subject of ongoing work.