  TACRED is one of the largest and most widely used sentence-level relation
extraction datasets. Proposed models that are evaluated using this dataset
consistently set new state-of-the-art performance. However, they still exhibit
large error rates despite leveraging external knowledge and unsupervised
pretraining on large text corpora. A recent study suggested that this may be
due to poor dataset quality. The study observed that over 50% of the most
challenging sentences from the development and test sets are incorrectly
labeled and account for an average drop of 8% f1-score in model performance.
However, this study was limited to a small biased sample of 5k (out of a total
of 106k) sentences, substantially restricting the generalizability and broader
implications of its findings. In this paper, we address these shortcomings by:
(i) performing a comprehensive study over the whole TACRED dataset, (ii)
proposing an improved crowdsourcing strategy and deploying it to re-annotate
the whole dataset, and (iii) performing a thorough analysis to understand how
correcting the TACRED annotations affects previously published results. After
verification, we observed that 23.9% of TACRED labels are incorrect. Moreover,
evaluating several models on our revised dataset yields an average f1-score
improvement of 14.3% and helps uncover significant relationships between the
different models (rather than simply offsetting or scaling their scores by a
constant factor). Finally, aside from our analysis we also release Re-TACRED, a
new completely re-annotated version of the TACRED dataset that can be used to
perform reliable evaluation of relation extraction models.
