90% F1 Score in Relation Triple Extraction: Is it Real? (2023)

Abstract: Extracting relational triples from text is a crucial task for constructing knowledge bases. Recent advancements in joint entity and relation extraction models have demonstrated remarkable F1 scores (≥ 90%) in accurately extracting relational triples from free text. However, these models have been evaluated under restrictive experimental settings and unrealistic datasets. They overlook sentences with zero triples (zerocardinality), thereby simplifying the task. In this paper, we present a benchmark study of state-of-the-art joint entity and relation extraction models under a more realistic setting. We include sentences that lack any triples in our experiments, providing a comprehensive evaluation. Our findings reveal a significant decline (approximately 10-15% in one dataset and 6-14% in another dataset) in the models’ F1 scores within this realistic experimental setup. Furthermore, we propose a two-step modeling approach that utilizes a simple BERT-based classifier. This approach leads to overall performance improvement in these models within the realistic experimental setting.

## introduction a crucial aspect of the relation extraction task involves the identification of sentences that lack any relational triples. this aspect naturally arises in real-world relation extraction scenarios. for instance, when extracting knowledge graph triples from online text, the majority of sentences may not mention any such triples. although this aspect has been explored in other nlp tasks, such as machine reading comprehension, where models should correctly identify when a given passage lacks an answer rather than providing an incorrect one @xcite @xcite , it has not received sufficient attention in recent relation extraction research. there are two distinct approaches for entity and relation extraction: classification approach and joint approach. in the classification approach @xcite @xcite @xcite , entities are already given and models focus on classifying the relations among pairs of entities. this approach includes sentences with zero triples in the experiments, where the relation among all entity pairs in such sentences is labeled as a 'none' relation. on the other hand, the joint extraction approach @xcite @xcite @xcite @xcite @xcite involves models extracting both entities and relations simultaneously. however, in this approach, sentences with zero triples are not considered in the experiments, which makes the task significantly easier. consequently, recent joint extraction models achieve exceptionally high f1 scores on benchmark datasets. in this study, our objective is to assess the performance of state-of-the-art relational triples extraction models when sentences with zero triples are included. to achieve this, we conduct comprehensive experiments using the widely used new york times (nyt) datasets. we evaluate a total of 9 recent state-of-the-art models in an end-to-end fashion. the results of our experiments reveal a significant decline in the performance of these models under this experimental setting. across all of the evaluated models, we observe an approximate drop of 10-15% in the f1 score in one dataset, and a drop of around 6-14% in another dataset. these findings highlight the challenges posed by sentences without triples and emphasize the need for improved approaches to handle such cases effectively. additionally, we have identified that sentences often contain clue tokens that can be leveraged to detect the presence of relations, even without identifying the corresponding entities. we include such examples in ## end-to-end modeling of relation extraction with zero-cardinality for our experiments, we select nine state-of-the-art joint entity and relation extraction models: ptr-net @xcite , tplinker @xcite , casrel @xcite , tdeer @xcite , prgc @xcite , pfn @xcite , grte @xcite , onerel @xcite , and birte @xcite . all of these models utilize bert @xcite as an encoder. for our experiments with the nyt24* dataset, where sentences are cased, we utilize the bert_base_cased model. on the other hand, for the nyt29* dataset, where sentences are uncased, we use the bert_base_uncased model. ptrnet @xcite adopts a sequenceto-sequence (seq2seq) approach, extracting triples uniformly regardless of the relations involved. the remaining models employ relation-specific sequence or matrix labeling methods to extract triples. originally, these models are trained solely on sentences containing one or more triples, excluding sentences with zero triples from their training and test datasets. however, we adapt these models to handle sentences with zero triples as well. in the case of sequence labeling or matrix labeling approaches, all tokens in the zero-cardinality sentences are labeled with the 'o' tag (representing the "other" tag). for sequence generation approaches (such as seq2seq), the decoder generates the "end of sequence" (eos) tag as the first token, indicating the absence of any relational triple in the sentence. below is a brief description of each of these models. we employ the same hyper-parameters as specified in their respective papers. ## two-step modeling of relation extraction with zero-cardinality we have observed that most relational triples in sentences are associated with specific clue tokens. while this may not always hold true due to the distant supervision used in creating the nyt datasets, it is applicable to many cases. we have included relevant examples in to accomplish this, we feed the sentences with a 'cls' prefix token (cls w 1 w 2 ..... w n ) into a pretrained bert_base model with a hidden dimension of h. we utilize the vector representation of the 'cls' token to determine whether the sentence contains any relational triples or not. we refer to this classifier as the zero-cardinality classifier (zcc). we explore two distinct approaches for this classifier: to begin, we train the classifier on the 'wz' training dataset, while training the joint extraction models on the 'nz' training set. during the inference phase, if the classifier model indicates the presence of triples in a test instance, we subsequently pass it to the joint extraction models to extract the exact triples. this two-step process enables us to effectively filter out sentences that do not contain any triples. we include the architecture of our proposed zerocardinality classifier in fig 1 . we use binary crossentropy loss and adamw @xcite optimizer to update the model parameters. we use mini-batch size of 16 and an early stop ## datasets preparation & evaluation metric the new york times (nyt) dataset holds significant importance as a benchmark for relation extraction. several studies @xcite @xcite ) utilize the derived nyt29 and nyt24 datasets, which originate from the original nyt10 @xcite and nyt11 @xcite training corpus, respectively. zeng et al. (2018 ); takanobu et al. (2019); nayak and ng (2020) exclude sentences without triples and partition the dataset into training, validation, and test sets. subsequent research papers @xcite @xcite @xcite @xcite build upon this modified version of the datasets, which is comparatively easier, and achieve exceptionally high f1 scores on these datasets. this trend reflects the prevalence of simplified datasets in recent works, potentially overestimating the performance of relation extraction models when faced with more realistic scenarios. in order to enhance the realism of the joint extraction task, we augment the nyt29 and nyt24 datasets by incorporating sentences with zero triples from the original nyt10 and nyt11 training corpus, respectively. these augmented datasets are referred to as nyt29* and nyt24* hereafter. the specifics regarding the training, validation, and test splits of the nyt24* and nyt29* datasets can be found in to evaluate the state-of-the-art (sota) models, we conduct experiments using two distinct training and test settings. these settings are as follows: by employing these two different experimental designs, we aim to gain insights into the robustness of the joint extraction models and their ability to handle different scenarios. ## results & discussion to begin our analysis, we assess the performance of state-of-the-art (sota) end-to-end models under the new experiment settings, which now include sentences with zero cardinality. the results of these experiments are presented in next, we proceed to train these models using the 'wz' sentences. upon analysis, we note that their performance on the 'nz' sentences experiences a decline of 4-8%, with the exception of the tdeer and casrel models. interestingly, the tdeer and casrel models exhibit comparable performance on the 'nz' test set, regardless of whether they were trained on 'nz' or 'wz' training data. however, the introduction of sentences with zero triples during the training process tends to confuse these models, leading to a negative impact on their recall. consequently, the models struggle to accurately extract valid triples due to the presence of such adversarial examples. furthermore, in this training setting, we observe an improvement of 2-8% in the models' performance on 'wz' sentences. nevertheless, the best f1 score reported on the stringent 'nz' test set for nyt24* is 0.926 (achieved by the grte model). in contrast, the best f1 score attained on the 'wz' test set for nyt24* is 0.82 (achieved by the birte model). this signifies a 10% drop in the best f1 score when transitioning to the experiment's more diverse setting. similarly, we observe a 6% decrease in the best achieved f1 scores on the 'wz' test set for nyt29* compared to the 'nz' test set. in the second step of our two-step approach, only the sentences predicted by the classification model to have existing triples are passed on to the triple extraction model. for this step, we train the stateof-the-art (sota) models exclusively on the 'nz' sentences to facilitate triple extraction. in our observations reveal an improvement of approximately ∼ 8% in the 'wz' sentences for both the nyt24* and nyt29* datasets when employing the two-step approach compared to the end-toend approach. specifically, for the nyt24* dataset, all sota models exhibit enhanced performance with the two-step approach over the end-to-end approach. however, for the nyt29* dataset, the performance is not consistently improved. in the case of four models (onerel, birte, prgc, and ptrnet), we observed a minor drop of up to ∼ 1% with the two-step approach. overall, we conclude that the two-step approach either improves the performance of these models or achieves competitive performance when compared to the end-to-end approach in this new experimental setting for relation extraction. ## conclusion in this work, we present an innovative and challenging experiment design for relation extraction, which incorporates sentences containing zero triples (referred to as zero-cardinal sentences) in the dataset. we conduct comprehensive experiments involving 9 state-of-the-art (sota) models using the widely-used new york times datasets. to tackle this task, we devise both an end-to-end modeling approach and a two-step modeling approach. during our investigations, we make a significant observation in the end-to-end modeling, where we notice a drop in the f1 score by approximately 10-15% and 6-14% in two versions of the nyt dataset. to address this issue, we propose the integration of a bert-based classifier as an additional step for this task. remarkably, this approach either achieves performance comparable to the end-toend approach or even surpasses it. we believe that our benchmark, focusing on relational triple extraction with zero-cardinality, will prove immensely valuable for future research in this domain. by introducing this unique experiment design, we aim to stimulate further advancements and foster progress in this field.