The State of Relation Extraction Data Quality: Is Bigger Always Better? (2024)

Abstract: Relation extraction (RE) extracts structured tuples of relationships (e.g. friend, enemy) between entities (e.g. Sherlock Holmes, John Watson) from text, with exciting potential applications. Hundreds of RE papers have been published in recent years; do their evaluation practices inform these goals? We review recent surveys and a sample of recent RE methods papers, compiling 38 datasets currently being used. Unfortunately, many have frequent label errors, and ones with known problems continue to be used. Many datasets focus on producing labels for a large number of relation types, often through error-prone annotation methods (e.g. distant supervision or crowdsourcing), and many recent papers rely exclusively on such datasets. We draw attention to a promising alternative: datasets with a small number of relations, often in specific domains like chemistry, finance, or biomedicine, where it is possible to obtain high quality expert annotations; such data can more realistically evaluate RE performance. The research community should consider more often using such resources.

## introduction relation extraction (re) methods extract tuple structures from unstructured text, where structures consist of types of relationships, e.g. is a member of an organization, and entities involved with them, e.g. samuel gompers, american federation of labor. @xcite ; @xcite ; @xcite ; @xcite ; @xcite claim that applications of these methods include populating knowledge graphs, and we find that realistic downstream applications also benefit from these methods: for example, automatic extraction of relations could replace manual extraction of {friend, enemy} relations in icelandic sagas @xcite , and could replace manual extraction of financial acquisition relations from news text in @xcite ; @xcite , helping to decrease labor in social science and humanities studies. however, significantly noisy dataset labels and evaluation on datasets using different performance metrics than intended may impact estimation of re method performance for a realistic setting. while some literature acknowledges the issues, various recent evaluations and efforts to construct datasets ignore them. we explore characteristics and application of 38 datasets for evaluating re methods. while our review is preliminary, we find some persistent patterns: although many evaluations are exclusively on datasets with larger numbers of relation types (f2), we encourage more of the research community to broaden evaluation to datasets with fewer relation types, given f3. evaluating on various datasets which are more likely to have expert-annotated labels and domain-specific text may have a higher chance to provide a more accurate estimation of re performance for a realistic setting, given f1. ## potential of relation extraction methods for realistic applications instead of automatically extracting relations, studies in various applications manually extract relations from text for downstream analysis. mac carron and @xcite compare social network structure in icelandic saga text and modern-day social networks by manually extracting friend and enemy relationships between saga characters. @xcite and @xcite use manually extracted financial acquisition relations from various news texts, even though automatic extraction has been investigated for these relations @xcite . ## disconnect between datasets and realistic application given promise of re for realistic applications ( §2), we ask: does performance on the datasets that re methods evaluate on give an accurate indication of how such methods will perform in a variety of realistic use cases? we explore, for datasets: (1) potential sources of error when assigning relation instance labels, (2) literature about labelling errors and evaluation on widely used datasets, (3) persistence of ignoring literature that addresses the issues. ## recommendations despite data quality challenges that may affect estimation of re method performance on realistic applications ( §3), we find potential for using re methods in real-world applications such as those described in §2. based on findings in §4, we provide two types of recommendations: (1) on selecting datasets to use for evaluation, and (2) on constructing future datasets to use for evaluation. on selecting datasets for evaluation. we encourage the research community to broaden evaluation to include datasets with smaller numbers of relation types, which are more likely to be annotated by experts, and ideally to use multiple such datasets. this helps test the flexibility of a method across diverse relation types and domains. to further strengthen confidence in dataset quality, researchers can also manually check correctness of a sample of labels on familiar relation types (if any), and check the literature for potential revised versions of a dataset. while we advocate for smaller and higher quality data, we note a counterargument that larger datasets-even with label noise-are crucial for training many relation extraction methods. however, the point of relation extraction research is to support applications, and we believe training data will be sparse and very expensive to obtain in most realistic settings, since annotations require significant domain expertise-heavy supervision is not a feasible modeling approach. therefore evaluation ought to be the primary role of relation extraction annotation, where more accurate labels are ideal, even if there are fewer of them. on future construction of datasets. while evaluation on noisy datasets may help to provide a rough indication of re method performance, noisy labels render datasets unhelpful for accurately estimating performance in a real world application. therefore, considering strategies to increase recall of labels assigned through distant supervision using approaches such as @xcite could help to label larger datasets more accurately and more efficiently-we find recent datasets @xcite aim to increase precision, but do not check or address recall of labels. further, defining relation types unambiguously is helpful for avoiding manual labelling issues such as in @xcite . ## conclusion relation extraction is a popular task with hundreds of relevant papers published in recent years. methods continue to improve and are becoming more promising for real world applications. first, we examined factors that potentially undermine relation extraction evaluation. next, we provided recommendations to overcome these challenges, involving broadening evaluation to include smaller and higher quality datasets, and considering strategies to increase the recall of labels in new datasets, to improve estimation of relation extraction performance for realistic settings.