Beyond the Numbers: Transparency in Relation Extraction Benchmark Creation and Leaderboards (2024)

Abstract: This paper investigates the transparency in the creation of benchmarks and the use of leaderboards for measuring progress in NLP, with a focus on the relation extraction (RE) task. Existing RE benchmarks often suffer from insufficient documentation, lacking crucial details such as data sources, inter-annotator agreement, the algorithms used for the selection of instances for datasets, and information on potential biases like dataset imbalance. Progress in RE is frequently measured by leaderboards that rank systems based on evaluation methods, typically limited to aggregate metrics like F1-score. However, the absence of detailed performance analysis beyond these metrics can obscure the true generalisation capabilities of models. Our analysis reveals that widely used RE benchmarks, such as TACRED and NYT, tend to be highly imbalanced and contain noisy labels. Moreover, the lack of class-based performance metrics fails to accurately reflect model performance across datasets with a large number of relation types. These limitations should be carefully considered when reporting progress in RE. While our discussion centers on the transparency of RE benchmarks and leaderboards, the observations we discuss are broadly applicable to other NLP tasks as well. Rather than undermining the significance and value of existing RE benchmarks and the development of new models, this paper advocates for improved documentation and more rigorous evaluation to advance the field.

## introduction we examine the transparency in benchmarks and leaderboards, focusing on the relation extraction (re) task. our analysis utilises two broadly accepted re datasets, tacred @xcite and nyt @xcite . while this paper focuses on the transparency of re benchmarks and leaderboards, the observations we discuss are also relevant to other areas of natural language processing (nlp). the development of state-of-the-art (sota) models in nlp is heavily reliant on benchmarks for evaluation. these benchmarks not only serve as a standard for assessing model performance but also play a pivotal role in shaping the perceived progress within the field. however, the current benchmarks often lack transparency in regard to their creation process, which can significantly impact the reliability of the evaluations conducted using them. opaque benchmarks and the absence of detailed performance analysis can obscure the true generalisation capabilities of models @xcite . when benchmarks are not fully transparent -lacking comprehensive metadata, clear articulation of limitations, and rigorous evaluation reports -their ability to accurately reflect a model's robustness and generalisability is compromised. this can lead to an overestimation of progress, as models may appear to perform well on certain benchmarks but fail to generalise effectively to different or more complex datasets. to enhance transparency and reproducibility in the evaluation of models, it is essential to publish the annotation guidelines and instructions that were provided to annotators during the creation of benchmarks. understanding the exact criteria and procedures used in annotation is critical for interpreting the results obtained from these benchmarks and for comparing the performance of different models. it is also important to recognise that widely used benchmarks such as tacred @xcite , tacred-re @xcite , and nyt @xcite cover only a subset of all possible relations. this limitation should be considered when evaluating models, as these benchmarks do not necessarily capture the full complexity of relation extraction task. additionally, when asserting that a new system outperforms existing ones, it is crucial to provide more granular results beyond aggregate metrics like weighted average or macro f1-score. these metrics, while useful, may not be sufficiently informative, particularly in the context of datasets with a large number of labels @xcite and significant class imbalances, such as nyt or tacred. although this position paper addresses certain issues with existing re benchmarks and evaluation approaches, it does not seek to diminish their significance or the value of developing new re models, which are crucial for advancing the nlp field. instead, it aims to promote improved documentation of benchmarks and the adoption of more rigorous evaluation practices for sota re systems. ## related work despite the critical role of data in nlp, the documentation of the creation process of existing datasets remains scarce, unstandardised, and often lacks transparency, even for publicly available datasets @xcite @xcite @xcite . @xcite addresses the issue of insufficient transparency in dataset creation by proposing that dataset creators accompany each dataset with a datasheet. this datasheet would document essential information about the dataset's creation process, thereby enhancing the reproducibility of machine learning experiments and helping to mitigate potential biases. they outline seven key stages of the dataset lifecycle: motivation, composition, collection process, preprocessing/cleaning/labeling, intended uses, distribution, and maintenance. the lack of transparency in benchmark creation significantly impacts the evaluation of models trained on these benchmarks. as kovatchev and lease, 2024 highlights, many evaluation frameworks operate under the implicit assumption that a particular dataset is representative of the task it is intended to benchmark. however, systematic approaches to testing model generalisation remain limited @xcite . to address this gap, @xcite propose the use of dataset similarity vectors, which consider various dimensions of the data, such as noise and ambiguity features, to more accurately predict the generalisation capabilities of models trained on these datasets. @xcite present a comprehensive taxonomy of methods for studying the generalisation capabilities of models and introduce the genbench evaluation card template foot_0 to assist researchers in systematically documenting, justifying, and tracing their generalisation experiments. evaluating the generalisation capabilities of models has become increasingly complex in the era of large language models (llms), which strive to achieve humanlike generalisation but are trained on vast, uncontrolled, and often nontransparent datasets. to enhance the transparency of model evaluation processes, researchers advocate for testing new sota models in challenging scenarios involving perturbed instances @xcite @xcite , thereby assessing model capabilities in more realistic settings than those provided by traditional test sets. linzen, 2020, when discussing the limitations of current evaluation approaches, particularly in the context of developing systems with human-like generalisation capabilities, introduces the generalisation leaderboards. these leaderboards evaluate systems on test sets derived from distributions different from those used during training. this approach addresses the limitation that testing a model on data drawn from the same distribution as the training set does not necessarily demonstrate the model's ability to effectively solve the task; rather, it may merely reflect the model's proficiency in capturing statistical patterns specific to the training data. in addition to traditional leaderboards, which often rank sota systems based solely on holistic metrics such as aggregate f1-score, @xcite propose leaderboards that incorporate more fine-grained metrics and offer functionality for direct analysis of misclassifications. this approach allows for a more detailed comparison of system performance, enabling users to directly identify the strengths and weaknesses of specific systems, thereby enhancing the transparency of leaderboards. ## transparency in benchmark creation current relation extraction benchmarks still lack transparency in their creation processes, making it difficult to assert that they generalise well on outof-distribution data. for instance, we often lack detailed information about the text sources used to create these benchmarks. transparency in the cre-ation of re datasets is crucial not only for mitigating potential biases but also for facilitating progress in the field. by better understanding the limitations of existing re benchmarks we are able to consequently better understand the limitations of systems that make use of these data. we examine the problem of lacking transparency through the lens of two of the most widely used general-purpose relation extraction benchmarks, namely nyt @xcite and tacred @xcite datasets. these benchmarks are broadly accepted by the nlp community and continue to be widely used, even in the era of llms @xcite @xcite @xcite . both the nyt and tacred datasets address the task of sentence-level relation extraction. ## transparency in leaderboard performance evaluation the transparency in the benchmark creation process has a direct impact on the ability to adequately evaluate the system trained on a dataset in question and therefore analyse its generalisation capabilities. one of the ways of measuring progress in particular nlp field are leaderboards. despite the fact that leaderboards push the nlp field forward, they also lack transparency on evaluation process and mostly are limited to the ranking based on holistic metrics such as accuracy or f1-score @xcite . for instance, both tacred foot_7 and nyt foot_8 leaderboards on a widely used platform paperswithcode rely on f1-score as a holisitic metric to rank the re models. moreover, not only traditional leaderboards lack fine-grained metrics in their ranking approach, but also the papers that report sota results follow this trend, which can lead to an emphasis on achieving top leaderboard positions rather than genuinely addressing the underlying task -a phenomenon known as sota-chasing @xcite . recent papers reporting sota-performance on nyt, tacred, and tacred-re foot_9 @xcite @xcite @xcite report only aggregate metrics such as micro or macro f1-score, recall, and precision. consequently, these evaluations lack more fine-grained, class-based metrics, which are crucial for the analysis of re systems dealing with a large number of relations such as the 42 labels in tacred and the 25 labels in nyt. in the context of imbalanced datasets like tacred and nyt, a system may achieve high overall metrics by always predicting a 'no_relation' class. however, this outcome does not indicate that the system has indeed effectively learned to solve the relation extraction task across a diverse set of over 20 labels. notably, even the original papers presenting tacred @xcite , tacred-re @xcite , nyt @xcite do not contain fine-grained, class-based metrics. given the significant class imbalance reflected in the tables 1 and 2 , as well as the fact that many relations in both tacred and nyt are represented by only a few instances, such as '/people/person/profession' in nyt (see benchmarks such as nyt, tacred, and tacred-re lack standardised guidelines for reporting results, leading to inconsistencies across publications that report sota results on these benchmarks @xcite . this lack of agreement can cause discrepancies in leaderboard rankings. for example, there is no consensus on which aggregated score should be used on platforms like paperswithcode. the current topperforming model on the tacred benchmark @xcite reports the micro-f1 score, which is also used for ranking. in contrast, the current second @xcite and third @xcite top-ranked models on tacred report the macro-f1 score, which is also utilised for their ranking on paperswithcode. this inconsistency in evaluation metrics raises concerns about the reliability of the leaderboard rankings. additionally, there is no overlap between the top-performing models listed on paperswithcode leaderboards for nyt and tacred, meaning that all top-performing models for tacred are different from those for nyt. this further complicates the analysis of these models' generalisation capabilities and makes it difficult to assess model ranking consistency across re benchmarks. focusing exclusively on achieving high performance on a single benchmark, without considering results across multiple benchmarks, can result in models that are overly specialised for specific benchmarks. this, however, does not necessarily indicate meaningful progress in addressing a particular nlp task @xcite , such as relation extraction. papers reporting sota results on re, including the original tacred @xcite , tacred-re @xcite , nyt @xcite , often do not provide information on whether the issue of class imbalance was addressed. such details should be included in system description papers, particularly when reporting new sota results. for instance, the authors of the biographical re dataset @xcite tackled the problem of large class imbalance by removing some of majority class relations, thereby equalising them with the sum of all other relations. model performance ceiling @xcite ) may be caused by the presence of noisy data, which can limit the potential for improvement by new re methods. as discussed in section 3, this noise can originate from both distantly-supervised datasets, such as nyt, and fully-supervised crowdsourced datasets, such as tacred. additionally, the way the task is formulated, whether as a single-label (tacred) or multi-label (nyt) classification task, can contribute to performance limitations. for example, an re model might make a correct prediction, but due to the task being framed as a singlelabel classification problem -despite real-world instances potentially containing multiple relations -this could lead to misclassification. such factors should be considered when reporting new sota results. moreover, in the era of llms, it is possible that multiple outputs generated by an llm for an re task could be correct @xcite , a nuance that is not captured by holistic metrics like aggregate f1-score. due to the mentioned limitations of traditional leaderboards such as the ones utilised on the paperswithcode platform, @xcite suggest an explainaboard interactive tool that provides both holistic and fine-grained metrics as well as functionality for direct analysis of misclassifications. such an extension of traditional leaderboards enables the direct detection of strengths and weaknesses of a particular system, as well as of a benchmark, thereby enhancing the ability to assess the generalisation capabilities of systems, such as those used for relation extraction. moreover, evaluating model performance on a test set drawn from the same distribution as the training set does not necessarily demonstrate a model's ability to solve an underlying task (linzen, 2020), such as relation extraction. to address this issue, linzen, 2020 propose generalisation leaderboards, which would evaluate systems on test sets derived from different distributions than the training set. for instance, it would be valuable to assess a system fine-tuned on tacred data for its ability to extract the same subset of relations present in the nyt dataset, as strong performance on one dataset does not necessarily indicate robust generalisation capabilities. additionally, techniques such as adversarial attacks @xcite @xcite can further test the true capabilities of re systems by exposing their vulnerabilities and resilience to challenging scenarios. ## conclusion and future work in this work, we have highlighted several limitations in the benchmark documentation and use of traditional leaderboards, particularly those employed for the relation extraction task. limitations in benchmark documentation include the absence of comprehensive descriptive metadata, such as the source of the data or details regarding interannotator agreement, as well as an absence of clear articulation of the dataset's inherent limitations, such as large class imbalances and potential noise. furthermore, there is often insufficient discussion on methods to mitigate these issues. evaluating systems based on these re benchmarks inherently necessitates addressing the problems associated with insufficient documentation of the benchmarks. for instance, traditional leaderboards, such as those on paperswithcode, that play a significant role in advancing nlp, typically rely on holistic metrics like f1-score. however, these metrics fail to capture the complexity of the relation extraction task, especially in scenarios involving a large number of labels and highly imbalanced datasets, such as tacred, where most instances belong to a 'no_relation' class. additionally, papers reporting new sota results on re benchmarks like nyt and tacred often focus exclusively on aggregate metrics, neglecting class-based metrics, which obscures the nuanced performance of models across different relation types. this paper does not intend to undermine the significance and value of existing benchmarks such as tacred or nyt, which are crucial for the evaluation of models in the field, as well as the development of new sota approaches. instead, given the evolving perspective on data used in training deep learning models, our objective is to propose avenues for improving the documentation of benchmark creation processes, which would in turn help to better assess the generalisation capabilities of re models. additionally, we also aim to motivate the adoption of more rigorous evaluation practices, encouraging researchers to move beyond the limited scope of only reporting metrics such as aggregate f1-score, precision, and recall. this is particularly important in nlp tasks such as relation extraction, where the complexity is exacerbated by the presence of a large number of relations. it is also crucial to recognise that high performance on a specific re benchmark, such as tacred, tacred-re, or nyt, reflects only a model's ability to handle a subset of all possible relations. furthermore, even if a system performs well on a given subset of relations, it may struggle significantly when extracting the same relations from out-of-distribution data. our focus should not solely be on the development of new approaches, but also on critically analysing our systems and recognising the limitations of the data used for their evaluation. this critical perspective is essential for advancing the field and ensuring that our models are robust and generalisable. finally, this work serves as a position paper that highlights several issues in the creation of re benchmarks and the practices surrounding leaderboard evaluations. we acknowledge the limitations of this work, particularly the lack of extensive quantitative evidence. in our future research, we aim to conduct a comprehensive cross-dataset evaluation of re systems on the benchmarks discussed. such an evaluation will provide empirical support for the concerns raised and offer a more reliable assessment of the generalisation capabilities of current re systems.