TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task (2020)

Abstract: TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE). But, even with recent advances in unsupervised pre-training and knowledge enhanced neural RE, models still show a high error rate. In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? And how do crowd annotations, dataset, and models contribute to this error rate? To answer these questions, we first validate the most challenging 5K examples in the development and test sets using trained annotators. We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled. On the relabeled test set the average F1 score of a large baseline model set improves from 62.1 to 70.1. After validation, we analyze misclassifications on the challenging instances, categorize them into linguistically motivated error groups, and verify the resulting error hypotheses on three state-of-the-art RE models. We show that two groups of ambiguous relations are responsible for most of the remaining errors and that models may adopt shallow heuristics on the dataset when entities are not masked.

## introduction relation extraction (re) is the task of extracting relationships between concepts and entities from text, where relations correspond to semantic categories such as per:spouse, org:founded by or org:subsidiaries (figure 1 ). this makes re a key part of many information extraction systems, and its performance determines the quality of extracted facts for knowledge base population @xcite , or the quality of answers in question answering systems @xcite . standard benchmarks such as semeval 2010 task 8 @xcite and the more recent tacred the sentence contains the relation org:subsidiaries between the head and tail organization entities 'aerolineas' and 'austral'. @xcite are essential to evaluate new re methods and their limitations, and to establish open challenges. tacred is one of the largest and most widely used re datasets. it contains more than 106k examples annotated by crowd workers. the methods best performing on the dataset use some form of pre-training to improve re performance: finetuning pre-trained language representations @xcite @xcite or integrating external knowledge during pre-training, e.g. via joint language modelling and linking on entity-linked text @xcite @xcite ; with the last two methods achieving a state-of-the-art performance of 71.5 f1. while this performance is impressive, the error rate of almost 30% is still high. the question we ask in this work is: is there still room for improvement, and can we identify the underlying factors that contribute to this error rate? we analyse this question from two separate viewpoints: (1) to what extent does the quality of crowd based annotations contribute to the error rate, and (2) what can be attributed to dataset and models? answers to these questions can provide insights for improving crowdsourced annotation in re, and suggest directions for future research. to answer the first question, we propose the following approach: we first rank examples in the development and test sets according to the misclas-sifications of 49 re models and select the top 5k instances for evaluation by our linguists. this procedure limits the manual effort to only the most challenging examples. we find that a large fraction of the examples are mislabeled by the crowd. our first contribution is therefore a extensively relabeled tacred development and test set. to answer the second question, we carry out two analyses: (1) we conduct a manual explorative analysis of model misclassifications on the most challenging test instances and categorize them into several linguistically motivated error categories; (2) we formulate these categories into testable hypotheses, which we can automatically validate on the full test set by adversarial rewriting -removing the suspected cause of error and observing the change in model prediction @xcite . we find that two groups of ambiguous relations are responsible for most of the remaining errors. the dataset also contains clues that are exploited by models without entity masking, e.g. to correctly classify relations even with limited access to the sentential context. we limit our analysis to tacred, but want to point out that our approach is applicable to other re datasets as well. we make the code of our analyses publicly available. foot_0 in summary, our main contributions in this paper are: ## the tacred dataset all relation labels were obtained by crowdsourcing, using amazon mechanical turk. crowd workers were shown the example text, with head (subject) and tail (object) mentions highlighted, and asked to select among a set of relation label suggestions, or to assign no relation. label suggestions were limited to relations compatible with the head and tail types. 5 the data quality is estimated as relatively high by @xcite , based on a manual verification of 300 randomly sampled examples (93.3% validated as correct). the inter-annotator kappa label agreement of crowd workers was moderate at κ = 0.54 for 761 randomly selected mention pairs. ## an analysis of tacred label errors in order to identify the impact of potentially noisy, crowd-generated labels on the observed model performance, we start with an analysis of tacred's label quality. we hypothesize that while comparatively untrained crowd workers may on average produce relatively good labels for easy relation mentions, e.g. those with obvious syntactic and/or lexical triggers, or unambiguous entity type signatures such as per:title, they may frequently err on challenging examples, e.g. highly ambiguous ones or relation types whose scope is not clearly defined. an analysis of the complete dataset using trained annotators would be prohibitively expensive. we therefore utilize a principled approach to selecting examples for manual analysis (section 3.1). based on the tac-kbp annotation guidelines, we then validate these examples (section 3.2), creating new dev and test splits where incorrect annotations made by crowd workers are revised (section 3.3). ## an analysis of model errors after revising the dataset, we focus on the two open questions: which of the remaining errors can be attributed to the models, and what are potential reasons for misclassifications? to answer these, we first create an annotation task instructing the linguists to annotate model misclassifications with their potential causes (section 4.1). we then categorize and analyze the causes and formulate testable hypotheses that can be automatically verified (section 4.2). for the automatic analysis, we implemented a baseline and three state-of-the-art models (section 4.3). ## model error and dataset analysis in this section, we present our analysis results, providing an answer to the question: which of the remaining errors can be attributed to the models, and what are the potential reasons for these errors? we first discuss the findings of our manual misclassification analysis (section 5.1), followed by the results of the automatic analysis (section 5.2). ## conclusion and future work in this paper, we conducted a thorough evaluation of the tacred re task. we validated the 5k most challenging examples in development and test set and showed that labeling is a major error source, accounting for 8% absolute f1 error on the test set. this clearly highlights the need for careful evaluation of development and test splits when creating datasets via crowdsourcing. to improve the evaluation accuracy and reliability of future re methods, we provide a revised, extensively relabeled tacred. in addition, we categorized model misclassifications into 9 common re error categories and observed that models are often unable to predict a relation, even if it is expressed explicitly. models also frequently do not recognize argument roles correctly, or ignore the sentential context. in an automated evaluation we verified our error hypotheses on the whole test split and showed that two groups of ambiguous relations are responsible for most of the remaining errors. we also showed that models adopt heuristics when en-tities are unmasked and proposed that evaluation metrics should consider an instance's difficulty.