What Do You Mean by Relation Extraction? A Survey on Datasets and Study on Scientific Relation Classification (2022)

Abstract: Over the last five years, research on Relation Extraction (RE) witnessed extensive progress with many new dataset releases. At the same time, setup clarity has decreased, contributing to increased difficulty of reliable empirical evaluation (Taillé et al., 2020). In this paper, we provide a comprehensive survey of RE datasets, and revisit the task definition and its adoption by the community. We find that cross-dataset and cross-domain setups are particularly lacking. We present an empirical study on scientific Relation Classification across two datasets. Despite large data overlap, our analysis reveals substantial discrepancies in annotation. Annotation discrepancies strongly impact Relation Classification performance, explaining large drops in cross-dataset evaluations. Variation within further sub-domains exists but impacts Relation Classification only to limited degrees. Overall, our study calls for more rigour in reporting setups in RE and evaluation across multiple test sets.

## introduction information extraction (ie) is a key step in natural language processing (nlp) to extract information, which is useful for question answering and knowledge base population, for example. relation extraction (re) is a specific case of ie @xcite with the focus on the identification of semantic relations between entities (see figure 1 ). the aim of the most typical re setup is the extraction of informative triples from texts. given a sequence of tokens [t 0 , t 1 ..., t n ] and two entities (spans), s @xmath0, . . . , t j ] and s @xmath1, . . . , t v ], re triples are in the form (s a , s b , r), where @xmath2. because of the directionality of the relations, (s b , s a , r) represents a different triple. we survey existing re datasets-outside the biomedical domain-with an additional focus on an entity-oriented approach to restricted-domain parsing is proposed. the task definition. 1 existing re surveys mainly focus on modeling techniques @xcite @xcite . to the best of our knowledge, we are the first to give a comprehensive overview of available re datasets. we also revisit re papers from the acl community over the last five years, to identify what part(s) of the task definition recent work focuses on. as it turns out, this is often not easy to determine, which makes fair evaluation difficult. we aim to shed light on such assumptions. 2 moreover, recent work in nlp has shown that single test splits and in-distribution evaluation overestimate generalization performance, arguing for the use of multiple test sets or split evaluation @xcite . while this direction has started to be followed by other nlp tasks @xcite @xcite @xcite , for re cross-dataset and cross-domain evaluation have received little attention. we explore this direction in the scientific domain and propose to study the possible presence of distinctive sub-domains @xcite . sub-domains are differences between subsets of a domain that may be expected to behave homogeneously. using two scientific datasets, we study to what degree: (a) they contain overlapping data; (b) their annotations differ; and (c) sub-domains impact relation classification (rc)-the task of classifying the relation type held between a pair of entities (details in section 3). the contributions of this paper are: ## relation extraction datasets survey re has been broadly studied in the last decades and many datasets were published. we survey widely used re datasets in chronological order, and broadly classify them into three domains based on the data source: (1) news and web, (2) scientific publications and (3) wikipedia. an overview of the datasets is given in another widely used dataset is the new york times (nyt) annotated corpus, 3 first presented by @xcite . it contains over 1.8 million articles by the nyt between 1987 and 2007. nyt has been created with a distant supervision approach @xcite , using freebase @xcite as knowledge base. two further versions of it followed recently: @xcite (nyt-h) and @xcite published manually annotated versions of the test set in order to perform a more accurate evaluation. since 2017, three re datasets in the scientific domain emerged, two of the three as semeval shared tasks. in semeval-2017 task this resulted in the tacred dataset with over 100k instances, which is particularly well-suited for neural models. @xcite used ta-cred to make a fsl rc dataset and compared it to fewrel 1.0 and fewrel 2.0, aiming at a more realistic scenario (i.e., non-uniform label distribution, inclusion of pronouns and common nouns). all datasets so far present a sentence level annotation. to address this, @xcite published docred, a document-level re dataset from wikipedia and wikidata. the difference with a traditional sentence-level corpus is that both the intraand inter-sentence relations are annotated, increasing the challenge level. in addition to re, docred annotates coreference chains. dwie by @xcite is another document-level dataset, specifically designed for multi-task ie (named entity recognition, coreference resolution, relation extraction, and entity linking). lastly, there are works focusing on creating datasets for specific re aspects. @xcite , for example, proposed a chinese documentlevel re dataset for hard cases in order to move towards even more challenging evaluation setups. ## the relation extraction task conceptually, re involves a pipeline of steps (see figure 2 ). starting from the raw text, the first step consists in identifying the entities and eventually assigning them a type. entities involve either nominals or named entities, and hence it is either named entity recognition (ner) or, more broadly, mention detection (md). 5 after entities are identified, approaches start to be more blurry as studies have approached re via different angles. one way is to take two steps, relation identification (ri) and subsequent relation classification (rc) @xcite , as illustrated in figure 2. this means to first identify from all the possible entity pairs the ones which are in some kind of relation via a binary classification task (ri). as the proportion of positive samples over the negative is usually extremely unbalanced towards the latter @xcite , a priori heuristics are generally applied to reduce the possible combinations (e.g., entity pairs involving distant entities, or entity type pairs not licensed by the relations are not even considered). the last step (rc) is usually a multi-class classification to assign a relation type r to the positive samples from the previous step. some studies merge ri and rc @xcite into one step, by adding a no-relation (no-rel) label. other studies instead reduce the task to rc, and assume there exists a relation between two entities and the task is to determine the type (without a no-rel label). regardless, ri is influenced by the rc setup: relations which are not in the rc label set are considered as negative samples in the ri phase. some studies address this approximation by distinguishing between the no-rel and the none-of-the-above (nota) relation @xcite . note that, in our definition, the nota label differs from no-rel in the sense that a relation holds between the two entities, but its type is not in the considered rc label set. 6 what do you mean by relation extraction? re studies rarely address the whole pipeline. we analyze all the acl papers published in the last five years which contain the relation extraction keyword in the title and determine which sub-task is performed (ner/md, ri, rc). the result shows that gold entities are usually assumed for re, presumably given the complexity of the ner/md task on its own. most importantly, for end-to-end models, recent work has shown that ablations for steps like ner are lacking @xcite . our analysis further shows that it is difficult to determine the ri setup. while rc is always performed, the situation is different for ri (or no-rel). sometimes ri is clearly not done (i.e., the paper assumes a scenario in which every instance contains at least one relation), but most of the times it is either not clear from the paper, or done in a simplified scenario (e.g., datasets which already clear out most of the no-rel entity pair instances). as this blurriness hampers fair evaluation, we propose that studies clearly state which step they include, i.e., whether the work focus is on rc, ri+rc or the full re pipeline and how special cases (no-rel and nota) are handled. these details are utterly important as they impact both model estimation and evaluation. pipeline or joint model? the traditional re pipeline is, by definition of pipeline, prone to error propagation by sub-tasks. joint entity and relation extraction approaches have been proposed in order to alleviate this problem @xcite @xcite @xcite . however, @xcite recently discussed the challenge of properly evaluating such complex models. they surveyed the evaluation metrics of recently published works on end-to-end re referring to the strict, boundaries, relaxed evaluation setting pro- relation extraction paper task performed ner/md ri rc 2021 wang et al. (2021) cui et al. (2021) tang et al. (2021) (?) xie et al. (2021) (?) tian et al. (2021) ma et al. (2021) mathur et al. (2021) yang et al. (2021) huang et al. (2021b) (?) huang et al. (2021a) (?) 2020 kruiper et al. (2020) nan et al. (2020) alt et al. (2020) yu et al. (2020) shahbazi et al. (2020) (?) pouran ben veyseh et al. (2020) 2019 trisedya et al. (2019) (?) guo et al. (2019) yao et al. (2019) zhu et al. (2019) li et al. (2019) (?) ye et al. (2019) fu et al. (2019) dixit and al-onaizan (2019) obamuyide and vlachos (2019) (?) 2018 christopoulou et al. (2018) phi et al. (2018) 2017 lin et al. (2017) (?) while some recent work shifts to joint models, it is still an open question which approach (joint or pipeline) is the most robust. @xcite found that when incorporating modern pretrained language models (e.g., bert) using separate encoders can surpass existing joint models. since the output label space is different, separate encoders could better capture distinct contextual information. at the moment it is not clear if one approach is more suitable than the other for re. for this reason and because of our final goal, which is a closer look to sub-domains in the scientific field, we follow the pipeline approach and, following most work from open issues to summarize, open issues are: 1) the unclarity of re setups, as illustrated in ## scientific domain data analysis in this section, we present the two english corpora involved in the experimental study (section 4.1), explain the label mapping adopted for the crossdataset experiments (section 4.2), discuss the overlap between the datasets and the annotation divergence between them (section 4.3), and introduce the sub-domains considered (section 4.4). ## conclusions we present a survey on datasets for re, revisit the task definition, and provide an empirical study on scientific rc. we observe a domain shift in re datasets, and a trend towards multilingual and fsl for re. our analysis shows that our surveyed acl re papers focus mostly on rc and assume gold entities. other steps are more blurry, concluding with a call for reporting re setups more clearly. as testing on only one dataset or domain bears risks of overestimation, we carry out a cross-dataset evaluation. despite large data overlaps, we find annotations to substantially differ, which impacts classification results. sub-domains extracted from meta-data instead only slightly impact performance. this finding on sub-domain variation is specific to the explored rc task on the scientific setup considered. our study contributes to the first of three cross-domain re setups we propose (section 2) to aid further work on generalization for re.