IEP ile: Unearthing Large Scale Schema-Conditioned Information Extraction Corpus (2024)

Abstract: Large Language Models (LLMs) demonstrate remarkable potential across various domains; however, they exhibit a significant performance gap in Information Extraction (IE). Note that high-quality instruction data is the vital key for enhancing the specific capabilities of LLMs, while current IE datasets tend to be small in scale, fragmented, and lack standardized schema. To this end, we introduce IEPile, a comprehensive bilingual (English and Chinese) IE instruction corpus, which contains approximately 0.32B tokens. We construct IEPile by collecting and cleaning 33 existing IE datasets, and introduce schema-based instruction generation to unearth a large-scale corpus. Experimentally, IEPile enhance the performance of LLMs for IE, with notable improvements in zero-shot generalization. We open-source the resource and pre-trained models, hoping to provide valuable support to the NLP community.

## introduction large language models (llms) have achieved significant breakthroughs in multiple natural language processing (nlp) tasks @xcite @xcite @xcite @xcite @xcite . however, recent studies @xcite @xcite @xcite @xcite @xcite indicate a significant performance gap in the task of information extraction (ie) when utilizing llms. @xcite further illustrate that the major reason may lie in limited highquality, large-scale data corpus. concretely, most ie datasets are often limited in size, scattered in distribution, and lack standardization in schema 1 . faced with these limitations, there is an urgent need to collect instruction data in a unified and automated manner to build a high-quality, large-scale ie corpus. to this end, we collect and clean various existing ie datasets to obtain a comprehensive bilingual ie instruction dataset named iepile foot_1 . during the corpus construction, we find existing methods for constructing ie instruction data suffer from two issues for generalizable ie: 1) schema query disparity: there may be inconsistency in the number of schema queries within instruction between training and evaluation which can harm model generalization; 2) semantic confusion: the co-occurrence of semantically similar schemas within instructions may confuse the model. thus, we introduce a schema-based instruction generation strategy. we first construct a hard negative schema dictionary to promote the more frequent occurrence of semantically similar schema in instructions. then, we introduce batched instruction generation, dynamically limiting the number of schemas queried in each instruction to split_num, which not only addresses the issue of performance degradation due to inconsistent numbers of schema queries during training and evaluation, but also enhances the robustness when dealing with semantically confusing schema. finally, we obtain iepile which contains approximately 0.32b tokens. by fine-tuning a selection of the latest prominent models @xcite @xcite on the iepile dataset, we show that llms with iepile can yield better zeroshot performance than baselines. this achievement not only verifies the effectiveness of the iepile dataset but also provides a framework for creating ie datasets in other domains. ## iepile in this section, we introduce the construction of iepile and provide details in appendix b. ## experiments based on iepile, we fine-tune several latest prominent models, then compare their zero-shot generalization capabilities against a range of baseline models. results of the full supervision evaluation and training details are described in appendix c.