Efficient Data Learning for Open Information Extraction with Pre-trained Language Models (2023)

Abstract: Open Information Extraction (OpenIE) is a fundamental yet challenging task in Natural Language Processing, which involves extracting all triples (subject, predicate, object) from a given sentence. While labelling-based methods have their merits, generation-based techniques offer unique advantages, such as the ability to generate tokens not present in the original sentence. However, these generation-based methods often require a significant amount of training data to learn the task form of OpenIE and substantial training time to overcome slow model convergence due to the order penalty. In this paper, we introduce a novel framework, OK-IE, that ingeniously transforms the task form of OpenIE into the pre-training task form of the T5 model, thereby reducing the need for extensive training data. Furthermore, we introduce an innovative concept of ‘anchors’ to control the sequence of model outputs, effectively eliminating the impact of order penalty on model convergence and significantly reducing training time. Experimental results indicate that, compared to previous SOTA methods, OK-IE requires only 1/100 of the training data (900 instances) and 1/120 of the training time (3 minutes) to achieve comparable results.

## introduction open information extraction @xcite is the task of transforming unstructured text into semi-structured text. given an input sentence, an open information extraction system can output a set of corresponding sentence with elements in a pre-defined structure @xcite . these output can be used as an additional source of information to augment other tasks, such as building semi-automated knowledge graph construction systems @xcite , question answering @xcite , trigger detection @xcite and so on. recently, neural-based openie methods can be broadly divided into two categories: labeling-based methods @xcite @xcite and generation-based methods @xcite . labelingbased methods are known for their fast inference speed but have a limitation in that they cannot label tokens that are not present in the input sentence. on the other hand, generation-based methods possess the capacity to produce supplementary tokens in order to fulfill syntactic and comprehensive semantic requirements, albeit necessitating extensive training data and time investment to learn the task format of openie and converge towards desirable outcomes. gen2oie @xcite has set a robust baseline for generation-based methods by modeling the openie task as a seq2seq problem using two t5 @xcite models. the first t5 model is employed to extract all predicates from the sentence, while the second t5 model takes a sequence concatenated from the sentence and an individual predicate as input, outputting the corresponding unique triple. while the geno2ie sets a strong baseline, there are still areas for improvement. we address these with our proposed ok-ie framework, an order agnostic knowledge embedded information extraction framework. considering the simplicity of predicate extraction from sentences, we retain the first stage of gen2oie and focus on optimizing the second stage, which is responsible for generating the complete triple. specifically, unlike gen2oie, ok-ie cleverly transforms openie into t5's pre-training task, span corruption. this allows the model to learn the task with less data, eliminating the need for additional training to adapt the language knowledge from span corruption pre-training to the seq2seq. in addition, the seq2seq presumes that only one sequence order is correct. this order information is not explicitly provided to the model, which must learn it on the effectiveness of our proposed ok-ie framework is exemplified particularly in low-resource scenarios on the carb @xcite benchmark, as illustrated in figure 1 . summarizing, our key contributions are: predominant solutions to the openie task fall into two categories: labeling-based and generationbased methods. this paper aims to address the excessive requirements of training data and time inherent in generation-based methods, and thus we will primarily focus on discussing generationbased techniques. labeling-based methods, while openie system f1 f1% imojie 53.5 99.07 openie6 52.7 97.6 multi 2 oie 52.5 97.2 igl-oie 52.4 97.03 cigl-oie 54.0 100 detie 52.1 96.4 gen2oie 51.1 94.6 ok-ie (ours) 53.2 98.5 generation-based approaches present an alternative perspective in openie, treating both the input sentences and output triples as sequences. for example, imojie @xcite , leveraging an lstm model, sequentially generates triples. it takes an input sentence and produces the first triple, then concatenates this generated triple with the original sentence to form a new sequence. the process continues iteratively until the system reaches an end token. gen2oie @xcite , a powerful baseline in our study, employs a dual t5 model setup. the first t5 model is responsible for generating all predicates from a given sentence. subsequently, the second t5 model accepts the concatenation of the sentence and an individual predicate as input, producing the corresponding triple for that predicate. however, these techniques generally require extensive training data and time to yield satisfying results. despite gen2oie's utilization of the pretrained language model t5 as its backbone, a considerable discrepancy exists between the seq2seq task and t5's pre-training task, span corruption. the model still necessitates an ample amount of data to learn this new task format. concurrently, in the context of triples, both (subject, predicate, object) and (predicate, object, subject) orders are valid. however, due to the auto-regressive characteristic inherent in sequence generation, training presumes only one correct generation order. moreover, this assumed order is implicit and not directly conveyed to the model. consequently, the model might generate a correctly composed triple in an order inconsistent with the provided ground truth, resulting in an additional order penalty. this discrepancy can lead to model confusion, slow convergence, and thus, requires substantial training time. ## method in addressing the aforementioned issues, our proposed framework, ok-ie, pivots around two core aspects: the transformation of task format and the control over generation order. to better control the generation order, we introduce anchors and merge them with sentinels. ## conclusion in this paper, we have presented ok-ie, an efficient framework for open information extraction. by effectively leveraging pre-trained language models and addressing the challenges of generation order, ok-ie overcomes the computational inefficiencies of previous systems. our empirical evaluations demonstrate that ok-ie not only achieves comparable results to existing systems, but does so with significantly fewer resources. this underlines its capability in resource-constrained environments. our research stands as a testament to the possibility of accomplishing efficient information extraction with reduced data and minimal training time, thereby presenting new pathways for future research within the openie landscape.