Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods (2022)

Abstract: The ability to extract entities and their relations from unstructured text is essential for the automated maintenance of large-scale knowledge graphs. To keep a knowledge graph up-to-date, an extractor needs not only the ability to recall the triples it encountered during training, but also the ability to extract the new triples from the context that it has never seen before. In this paper, we show that although existing extraction models are able to easily memorize and recall already seen triples, they cannot generalize effectively for unseen triples. This alarming observation was previously unknown due to the composition of the test sets of the go-to benchmark datasets, which turns out to contain only 2% unseen data, rendering them incapable to measure the generalization performance. To separately measure the generalization performance from the memorization performance, we emphasize unseen data by rearranging datasets, sifting out training instances, or augmenting test sets. In addition to that, we present a simple yet effective augmentation technique to promote generalization of existing extraction models, and experimentally confirm that the proposed method can significantly increase the generalization performance of existing models.

## introduction relational triple extraction (rte), a more generalized version of relation extraction, is the task of extracting all relational triples in the form of (subject, relation, object) from a given sentence. the ability to extract such triples is much required in the construction and maintenance of knowledge graphs such as dbpedia @xcite , freebase @xcite , and wikidata @xcite ) from documents containing a large number of new and emerging information. with language model pretraining @xcite , rte methods achieved a new state-of-the-art @xcite @xcite . however, whether the performance of these methods attributes to their capabilities of recalling already seen data or their ability to generalize and extract relations from unseen data is yet to be scrutinized. to separately evaluate memorization and generalization, we categorize the triples in the test set into three types: entirely seen (completely overlaps with triples in their respective training sets), partially seen (overlaps partially), and unseen (completely new). we analyze common rte benchmark datasets nyt @xcite and webnlg @xcite using these categories, and find that 89.61% and 91.10% of triples in nyt and webnlg test sets are of the entirely seen type. this suggests that benchmark results on these datasets are heavily biased towards recalling seen data. thus, more reliable systematic evaluation methods are in need to test generalization performance. in this paper, we propose three natural strategies for evaluating generalization performance from a limited number of given partially seen and unseen triples. for the first two strategies, we directly increase the proportion of partially seen and unseen triples in test sets by 1) rearranging their respective datasets or 2) sifting out instances in their respective training sets that overlap with the test set, rendering them unobserved. for the last strategy, we 3) augment test sets by replacing entities in each test instance with similar (and probably not pre-observed) words in order to increase diversity as well as the proportion of partially seen and unseen triples. in addition to evaluating recent rte methods with the above evaluation strategies, we propose a simple yet effective augmentation technique called entity noising to help rte methods to generalize beyond training data. our contributions are: ## fine-grained re-evaluation of the current state-of-the-arts in this section, we mainly scrutinize the generalization capabilities of current relational triple extraction (rte) methods and show for the first time that they indeed struggle in extracting relational triples from the context for unseen cases. ## evaluating generalization performance as shown in ## entity noising we further propose entity noising, a simple augmentation technique to enhance the generalization performance of existing relational triple extraction methods. the key idea of entity noising is to replace the entities in the given training input sentence with completely random noisy words. to apply entity noising, we sample a random noisy word w ′ for each entity w, i.e., w ′ ∼ p (w ′ | w). the sampling strategy is defined as follows. first, with sampling strategy w ′ ∼ p (w ′ | w), entity noising is applied to a given training sentence finally, we determine which input x is fed to the extractor model with probability p (@xmath0. an overview illustration of entity noising is shown in figure 1 . entity noising is different from a commonly used data augmentation method such as @xcite which replaces entities with words similar to them. entity noising replaces entities with completely random noisy words. this feature allows the model to utilize entity-agnostic information, so that the model can learn to extract triples from sentences by focusing on the context information rather than the entities themselves. therefore, with entity noising, the model is kept away from memorizing the entity pair along with its relation. ## related work open information extraction (open ie) open ie is the task of extracting relations from the given text without predefined relation type @xcite @xcite . although open ie is a more general task than relational triple extraction, it is necessary to extract information using fixed relation type to get high quality relational triples from specific domains such as science and business. ## conclusion in this paper, we disclosed for the first time that recent relational triple extraction (rte) methods struggle to extract triples from unseen data, which was previously unknown due to the test-train overlap problem in popular benchmark datasets. to properly assess the generalization capabilities of rte methods, we developed three strategies to construct rearranged dataset, overlap sifted dataset, and augmented test set from original datasets. furthermore, we proposed a simple yet effective noising method to promote generalization and experimentally confirm that it effectively improves the generalization capabilities of existing rte methods.