C ross RE : A Cross-Domain Dataset for Relation Extraction (2022)

Abstract: Relation Extraction (RE) has attracted increasing attention, but current RE evaluation is limited to in-domain evaluation setups. Little is known on how well a RE system fares in challenging, but realistic out-of-distribution evaluation setups. To address this gap, we propose CrossRE, a new, freely-available cross-domain benchmark for RE, which comprises six distinct text domains and includes multi-label annotations. An additional innovation is that we release meta-data collected during annotation, to include explanations and flags of difficult instances. We provide an empirical evaluation with a state-of-the-art model for relation classification. As the meta-data enables us to shed new light on the state-of-the-art model, we provide a comprehensive analysis on the impact of difficult cases and find correlations between model and human annotations. Overall, our empirical investigation highlights the difficulty of cross-domain RE. We release our dataset, to spur more research in this direction.

## introduction relation extraction (re) is the task of extracting structured knowledge from unstructured text. although the fact that the task has attracted increasing attention in recent years, there is still a large gap in comprehensive evaluation of such systems which include out-of-domain setups @xcite . despite the drought of research on cross-domain evaluation of re, its practical importance remains. given the wide range of applications for re to downstream tasks which can vary from question answering, to knowledge-base population, to summarization, and to all kind of other tasks which require extracting structured information from unstructured text, out-of-domain generalization capabilities are extremely beneficial. it is essential to build re models that transfer well 1 https://github.com/mainlp/crossre he then went to live at chalcedon, whence in 367 he was banished to mauretania for harbouring the rebel procopius. to new unseen domains, which can be learned from limited data, and work well even on data for which new relations or entity types have to be recognized. one direction which is gaining attention is to study re systems under the assumption that new relation types have to be learned from few examples (few-shot learning; @xcite . one other direction is to study how sensitive a re system is under the assumption that the input text features change (domain shift; @xcite . there exists a limited amount of studies that focus on the latter aspect, and-to the best of our knowledge-there exists only one paper that proposes to study both, few-shot relation classification under domain shift @xcite . however, this last work considers only two domains-wikipedia text for training and biomedical literature for testing-and has been criticized for its unrealistic setup @xcite . in this paper, we propose crossre, a new challenging cross-domain evaluation benchmark for re for english (samples in figure 1 ). crossre is manually curated with hand-annotated relations covering up to 17 types, and includes multi-label annotations. it contains six diverse text domains, namely: news, literature, natural sciences, music, politics and artificial intelligence. one of the challenges of crossre is that both entities and relation type distributions vary considerably across domains. crossre is heavily inspired by cross-ner @xcite , a recently proposed challenging benchmark for named entity recognition (ner). we extend crossner to re and collect additional meta-data including explanations and flags of difficult instances. to the best of our knowledge, crossre is the most diverse re datasets available to date, enabling research on domain adaptation and few-shot learning. in this paper we contribute: ## related work despite the popularity of the re task (e.g. @xcite @xcite @xcite , the cross-domain direction has not been widely explored. there are only two datasets which can be considered an initial step towards cross-domain re. the ace dataset @xcite has been analyzed considering its five domains: news (broadcast news, newswire), weblogs, telephone conversations, usenet and broadcast conversations @xcite @xcite . in contrast to ace, the domains in crossre are more distinctive, with specific and more diverse entity types in each of them. more recently, the fewrel 2.0 dataset @xcite , has been published. it builds upon the original fewrel dataset @xcite -collected from wikipedia-and adds a new test set in the biomedical domain, collected from pubmed. ## baseline experiments we provide the evaluation of a state-of-the-art model on the proposed dataset. to establish baselines, we train models over each of the proposed domains. two major challenges affecting the dataset are the multi-label annotation setup and the highly sparse label distribution distinctive of each domain. ## meta-data analysis in this section, we use the meta-data collected during the annotation of the dataset for further analysis. we consider syntax ambiguity (sa) and un-certainty of the annotator (un) and examine the performance of our baseline model on such instances. with the low amount of instances in politics an literature, results are less pronounced and differences with the overall scores are absent in most cases. therefore, we focus here on the remaining three domains-natural science , music , ai . we observe slightly but consistently higher scores when taking out the cases marked with un, showing that they are challenging not only for the human but also the system. those are the cases identified as most challenging, specially considering the annotator's background (i.e. natural science and ai, mostly on cause-effect, part-of, us-age). the results in respect to the sa annotations are mixed: there is not a unified trend over domains or metrics. we attribute this to the fact that our model does not explicitly build upon syntactic features (e.g. syntactic trees; @xcite . finally, the scores from the data which consider the combination of sa and un increase over the baseline in the science domain, where taking out both sa and un individually increase over the all setup. in the music domain, where sa are frequent, excluding them result in a little drop of micro-f1 (75.96→74.67). in fact, the model is good on sa in the music domain: the majority of cases are on the general-affiliation label, which achieves high per-label f1 (87.04). we attribute it to the fact that in this domain there are many lists of entities and relative attributes, which structurally can be ambiguous, but often involve a similar relation structure. ai presents a similar trend as music, but the scores from the combination of sa and un increase a bit in both metrics. in conclusion, we do gain informative insights from the collected meta-data-especially when the annotator is unsure about the annotated relation, and also to understand whether syntactic ambiguity detected by the annotator impacts system accuracy. ## conclusion we present crossre, a new challenging manually-curated corpus for re. it is the first dataset for re covering six diverse text domains (news, politics, natural science, music, literature, ai) with annotations spanning 17 relation types. some annotations are enriched with meta-data information (explanation for the choice of the assigned label, identification of syntax ambiguity, and uncertainty of the annotator). throughout the annotation process and in the empirical validation, this meta-data proves to be useful and insightful. as it aids the analysis of the provided baseline, we invite the research community to both collect and release such additional information. we perform an empirical evaluation of crossre by applying state-of-the-art rc methods @xcite , and show the challenges of its highly imbalanced label distributions over the domains. the cross-domain dimension is currently underexplored in the re field. with this dataset we invite future work on cross-domain re evaluation, the exploration of domain-adaptive techniques (e.g. dapt; @xcite and other adaptation methods to improve the baseline set out in this work for the different data domains.