PIVOINE : Instruction Tuning for Open-world Entity Profiling (2023)

Abstract: This work considers the problem of Open-world Entity Profiling, a sub-domain of Open-world Information Extraction (Open-world IE). Unlike the conventional closed-world IE, Open-world IE is considered a more general situation where entities and relations could be beyond a predefined ontology. We seek to develop a large language model (LLM) that can perform Open-world Entity Profiling with instruction tuning to extract desirable entity profiles characterized by (possibly fine-grained) natural language instructions. In particular, we construct INSTRUCTOPENWIKI, a substantial instruction-tuning dataset for Open-world Entity Profiling enriched with a comprehensive corpus, extensive annotations, and diverse instructions. We finetune pretrained BLOOM models on INSTRUCTOPENWIKI and obtain PIVOINE, an LLM for Open-world Entity Profiling with strong instruction-following capabilities. Our experiments demonstrate that PIVOINE significantly outperforms traditional methods and ChatGPT-based baselines, displaying impressive generalization capabilities on both unseen instructions and out-of-ontology cases. Consequently, PIVOINE emerges as a promising solution to tackle the open-world challenge of entity profiling.

## introduction information extraction (ie) aims to discern meaningful information from unstructured data sources @xcite . a traditional ie pipeline contains an array of tasks, which include, but are not limited to, named entity recognition (ner) @xcite , entity linking (el) @xcite , entity typing (et) @xcite , relation extraction (re) @xcite , etc. ie plays a vital role in knowledge graph construction @xcite , search engine @xcite , and document analysis @xcite @xcite . most existing ie methods center around a closedworld setting with predefined ontologies. for instance, ner generally extracts named entities within several categories @xcite ; el focuses on associating mentions with a predefined ontology @xcite . to better break this limitation, we introduce open-world entity profiling, a series of entity-centric tasks within openworld information extraction (open-world ie) to accommodate broad and diverse requests related to entity profiles surpassing predefined ontologies' limits. past research on open-setting ie has predominantly focused on individual tasks of ie, such as el @xcite and openie @xcite . consequently, a noticeable gap exists in comprehensive end-to-end studies aiming to create more extensive entity profiles within an open-world setting. furthermore, closed-world ie usually extracts all information without focusing on desired targets. therefore, we design open-world entity profiling accepting an unstructured corpus and an instruction that characterizes target entities, identifies all entities within the context, and generates entity profiles, as shown in fig. 1 . with the emergence of large language models (llms) @xcite , generative ie based on llms holds substantial promise in addressing the open-world challenge, given their exceptional generalization capabilities on various instructions. open-world entity profiling can also be a pivotal capability for integrating plugins into the chatgpt system since it provides a flexible communication interface between llms and their plugins. nevertheless, existing research on llms reveals that they typically perform poorly in ie under zero-shot scenarios, necessitating appropriate instruction tuning to enhance their ie capabilities @xcite . therefore, instruction tuning @xcite becomes critical in endowing llms with open-world entity profiling abilities. to generalize well on unseen instructions and ontologies, we develop pivoine (instructionfollowing open-world information extraction). pivoine is an llm designed for open-world entity profiling. we formulate open-world entity profiling as an instruction-following auto-regressive generation task to generate comprehensive entity profiles in json. we explore eight popular categories of instructions in various granularities. each category imposes specific constraints on candidate entities. in pursuit of generalization over unseen cases, we develop an instruction tuning dataset instructopenwiki for open-world entity profiling, including diverse instructions that endow pivoine with solid instruction-following capability. instructopenwiki incorporates rich annotations, various instructions, and a delicate out-ofontology evaluation set, significantly contributing to the generalization of unseen cases. the contributions of this work are three-fold. first, we define open-world entity profiling and develop pivoine, which performs entity-centric ie without the limitations of predefined ontology. this flexibility allows for its generalization abilities and application across diverse downstream scenarios. second, we construct a substantial open-world entity profiling dataset instructopenwiki. third, we explore a comprehensive evaluation for openworld entity profiling. we meticulously design an open-world evaluation set incorporated in in-structopenwiki to assess open-world capabilities thoroughly, focusing on the generalization of unseen instructions and out-of-ontology entities. our contributions are verified with experiments and multifaceted analysis. most notably, pivoine exhibits impressive generalization capabilities on unseen instructions and out-of-ontology cases, demonstrating its robust potential to address the open-world challenge effectively. instruction tuning. instruction tuning is an emergent paradigm that finetunes llms on datasets described by instructions. @xcite finetune an llm with 175 billion parameters on various nlp datasets with instruction templates and proof instruction tuning can significantly improve zero-shot performance. @xcite show supervised instruction tuning and finetuning with human feedback helps llms align with human intent. this work is further extended by openai and becomes the product chatgpt foot_0 used as a baseline in our work. in this work, we create an instructionfollowing dataset instructopenwiki for openworld ie and employ instruction tuning to empower llms with open-world entity profiling abilities. ## methods we describe pivoine, an open-world entity profiling method with instruction-following abilities. we will introduce preliminaries ( §3.1), instruction-following open-world entity profiling ( §3.2), and construction of the dataset in-structopenwiki ( §3.3). ## evaluation a well-designed open-world evaluation set is essential for evaluating open-world entity profiling methods without bias. so we create an open-world evaluation set with rich out-of-ontology cases and design metrics for evaluating such performance. open-world evaluation set. previous work constructs open-world test sets by simply holding out a portion of entities from the training ontology. however, such methods may introduce potential risks and lead to insufficient evaluation. first, holding out entities from the training ontology also removes corresponding links in the training corpus, hindering the completeness of annotations. moreover, mentions of held-out entities still frequently appear in the training corpus even if they are not explicitly annotated, resulting in potential data leakage. therefore, we propose a delicate method that uses the time difference between wiki dumps to construct a strictly open-world test set. as shown in the bottom of fig. 3 , we use the wikidata dump on 05/30/2022 and the wikipedia dump on 06/20/2022 to build the training set. as the evaluation corpus, we filter all new articles between two wikipedia dumps, 06/20/2022 and 03/01/2023. and we select new entities appearing in the wikidata dump on 03/01/2023 but not presented in the wikidata on 05/30/2022 as out-of-ontology entities foot_3 . using time difference to build an out-of-ontology evaluation set minimizes potential data leakage and maximizes completeness of training annotations. ## experiments we carry out a comprehensive evaluation of pivoine, including experimental setup ( §5.1), main results ( §5.2), and further analysis ( §5.3). ## conclusion we propose open-world entity profiling, a challenging task that aims to extract out-of-ontology entity profiles with instructions. towards this grand mission, we create a large-scale open-world entity profiling dataset instructopenwiki and develop pivoine by instruction tuning. we conduct extensive experiments on diverse instruction categories and different model scales, showing pivoine is a trustworthy llm capable of following (possibly unseen) instructions in various granularities and extracting out-of-ontology entity profiles. valuable future works include extending pivoine to a larger scale and exploring more comprehensive instruction categories.