Universal Information Extraction with Meta-Pretrained Self-Retrieval (2023)

Abstract: Universal Information Extraction (Universal IE) aims to solve different extraction tasks in a uniform text-to-structure generation manner. Such a generation procedure tends to struggle when there exist complex information structures to be extracted. Retrieving knowledge from external knowledge bases may help models to overcome this problem but it is impossible to construct a knowledge base suitable for various IE tasks. Inspired by the fact that large amount of knowledge are stored in the pretrained language models (PLM) and can be retrieved explicitly, in this paper, we propose MetaRetriever to retrieve task-specific knowledge from PLMs to enhance universal IE. As different IE tasks need different knowledge, we further propose a Meta-Pretraining Algorithm which allows MetaRetriever to quicktly achieve maximum task-specific retrieval performance when fine-tuning on downstream IE tasks. Experimental results show that MetaRetriever achieves the new state-of-the-art on 4 IE tasks, 12 datasets under fully-supervised, low-resource and few-shot scenarios.

## introduction information extraction (ie) is the task of extracting specific information structures of interest from unstructured text @xcite . the various ie tasks at hand can be highly heterogeneous, as each task may have different extraction targets, such as entities @xcite , relations @xcite , and events @xcite . traditional ie methods tend to be task-specific, resulting in specialized and isolated approaches for different tasks. in recent years, researchers have made efforts towards universal information extraction, or universal ie @xcite , which aims to develop * corresponding author a unified method capable of solving a range of ie tasks. one proposed approach involves the use of a structure extraction language to express different extraction targets in a unified structure form, and pretrains a transformer model to generate this unified structure through a sequence-to-sequence process. however, this structure generation process can be limited in its ability to fully utilize contextual semantic correlation between the extracted information, as the transformer generation process is unidirectional. in cases where the information structure to be extracted is complex, the universal ie model may struggle to generate accurate results. previous studies on retrieval-augmented generation @xcite @xcite have shown that utilizing external knowledge bases to retrieve taskspecific information can improve the ability of models to generate complex sequences. however, this approach is not practical for universal ie, as it is almost impossible to build a knowledge base suitable for various ie tasks, and it is also the ultimate goal of ie. recently, research on knowledge probing @xcite @xcite has demonstrated that large amounts of knowledge are stored in pretrained language models (plms) and can be retrieved explicitly. utilizing this knowledge to enhance ie models could avoid the need for extensive resource construction. in light of these findings, the question arises: can plms be used as knowledge bases to retrieve knowledge and improve universal ie models? if so, universal ie models would be able to generate more accurate results. based on this idea, in this paper, we propose metaretriever, a pretrained model for universal ie. unlike existing method which generates results in a single step, metaretriever utilizes a retrieve-then-extract approach: first, it retrieves task-specific knowledge from itself, and then it uses this knowledge as additional input to extract information. however, different ie tasks require different knowledge, and simply retrieving information from plm can lead to irrelevant information being included, which can negatively impact performance. to address this, incorporating metalearning techniques @xcite , we develop a meta-pretraining algorithm (mpa) which allows the model to quickly learn the semantics of extraction targets of different tasks to retrieve the relevant knowledge. specifically, mpa pretrains the model with a bi-level optimization, in which the model are optimized to achieve maximal task-specific retrieval performance given the extraction targets after the parameters have been updated through a small number of gradient steps on a new ie task. thus, in the fine-tuning phase, metaretriever can learn downstream tasks quickly to retrieve relevant knowledge and disregard irrelevant information. we conduct experiments on 12 datasets of 4 main ie tasks under fully-supervised, few-shot and low-resource scenarios. experiments show that metaretriever significantly surpasses previous works and achieves the new state-of-the-art. on fully-supervised scenario, metaretriever achieves 0.7% gains on average over uie @xcite . on few-shot and low-resource scenarios, our model outperforms uie with over 2.0% improvement. our contributions can be summarized as follows: (1) we introduce a retrieve-then-extract perspective into universal ie. it retrieves task-specific knowledge from the pretrained language model to improve performance. (2) we propose a metapretraining algorithm to make models fast-adapt ## methodology our proposed metaretriever achieves universal ie via a retrieve-then-extract manner. it first retrieves task-specific knowledge from itself according to the input text and then generates the extraction results based on the retrieved knowledge. to make metaretriever only retrieve task-specific knowledge and neglect other distracting knowledge, we further proposed a meta-pretraining algorithm based on meta-learning technique. in this section, we first introduce how metaretriever works via the retrieve-then-extract manner and then detail the meta-pretraining algorithm. ## experiments 4.1 pretraining datasets to pretrain metaretriever, we collect a large-scale corpus using crocodile(cabot and navigli, 2021), an automatic relation extraction dataset construction tool. crocodile is based on the distant supervision technique @xcite which aligns texts from english wikipedia foot_1 and knowledge base from wikidata foot_2 to create distant labeled relation extraction data. since distant supervision inevitably introduces noise @xcite , crocodile further uses natural language inference (nli) to filter data with low confidence. finally, we construct a 71 million distant labeled pretraining corpus and obtained 6.9 million high-quality pretraining corpus after nli filtering. detailed statistics of our pretraining corpus are listed in appendix a. our analysis experiment (section 5.3) indicates that the pretraining corpus we constructed can yield comparable results in the trained model as the data constructed by uie. as uie's pretraining data is not open-sourced, our data will be the first 10million-level corpus to be open-sourced, and will be released after the acceptance of the paper. ## related work information extraction (ie) aims at extracting text spans or a tuple of text spans of interests from plain texts. there exist many specific ie tasks: named entity recognition @xcite , relational triple extraction @xcite , event extraction @xcite @xcite , etc. for a long time, researchers are devoted to propose task-customized and isolated methods to achieve these different ie tasks. in the recent years, with the rising of pretraining technique, pretraining a universal model for several nlp tasks has attracted a lot of attention @xcite @xcite @xcite . following this trend, several works make attempts to unify diverse ie tasks. @xcite first propose a span-offset-based generation to solve various ner tasks in a universal manner. @xcite use a similar way to solve different aspectbased sentiment analysis tasks. @xcite propose uie which designs structured extraction language to formulate all ie tasks in a unified form but it will experience poor performance when there exists complex structure to be extracted. several works have shown that retrieving knowledge from external knowledge bases can empower models to generate complex sequences @xcite @xcite and a large number of knowledge stored in pretrained language models can be "retrieved" explicitly @xcite @xcite . in this paper, we propose metaretriever to retrieve task-specific knowledge from plms to enhance universal ie. meta-learning aims to learn better learning algorithms from a series of tasks, i.e., learning to learn. in meta-learning, there exist diverse branches focusing on different aspects of the learning algorithm: learning-to-initialize @xcite , learning-to-compare @xcite , learning-to-optimize @xcite , etc. existing works are most devoted to solve the few-shot problem, cross-domain problem, etc. we make the first attempt to adapt meta-learning into universal ie. ## conclusion in this work, we present metaretriever, a novel approach for enhancing universal information extraction through the retrieval of task-specific knowledge from within the model itself. to ensure that the retrieved knowledge is truly task-specific and to minimize the impact of irrelevant or "noise" knowledge, we also propose a meta-pretraining algorithm that enables metaretriever to quickly adapt to downstream information extraction tasks. empirical results demonstrate the effectiveness of our approach, with metaretriever achieving new state-of-the-art performance on a range of information extraction tasks across 12 datasets in fullysupervised, low-resource, and few-shot scenarios. further analysis experiments also validate the contributions and effectiveness of metaretriever.