Instruct and Extract: Instruction Tuning for On-Demand Information Extraction (2023)

Abstract: Large language models with instruction-following capabilities open the door to a wider group of users. However, when it comes to information extraction – a classic task in natural language processing – most task-specific systems cannot align well with long-tail ad hoc extraction use cases for non-expert users. To address this, we propose a novel paradigm, termed On-Demand Information Extraction, to fulfill the personalized demands of real-world users. Our task aims to follow the instructions to extract the desired content from the associated text and present it in a structured tabular format. The table headers can either be user-specified or inferred contextually by the model. To facilitate research in this emerging area, we present a benchmark named InstructIE, inclusive of both automatically generated training data, as well as the human-annotated test set. Building on InstructIE, we further develop an On-Demand Information Extractor, ODIE. Comprehensive evaluations on our benchmark reveal that ODIE substantially outperforms the existing open-source models of similar size.

## introduction information extraction has conventionally been divided into a set of well-defined sub-tasks, including named entity recognition @xcite @xcite , relation extraction @xcite @xcite , event extraction @xcite @xcite and so on. these components serve as the basis for building complex systems such as virtual assistants and news monitoring systems. however, an average user might also have information extraction needs (as shown in figure 1 ) that do not align well with any of the well-defined tasks: the information elements (shape and taste) might not instruction: i personally prefer eating fruits. extract some key features of the fruits. ## instructie dataset in this section, we first formulate the on-demand information extraction task. following this, we delve into the details of automatically generating training data for instructie, as well as the process of human annotations for the test set. ## conclusion this paper introduces a new task, on-demand information extraction to fulfill users' personalized needs by extracting content based on instructions and organizing it in a table with user-specified or model-inferred headers. to benchmark this new task, we construct a comprehensive dataset in-structie including synthesized training data and human-annotated test data. our developed model odie outperforms existing open-source models in extensive experiments. limitations while this paper contributes to the research field by introducing the on-demand information extraction task and constructing the instructie dataset, it still has the following limitations: 1. model size and data constraints: the experiments presented in this paper primarily focus on the utilization of the 7b model. due to the limited computing resources, an exploration into the impact of varying model sizes and the potential benefits of using larger datasets could not be conducted. it remains an open question how scalability in terms of model size and data volume would affect the performance and efficiency of the on-demand information extraction task. 2. combination of direct and cot methods: in our experimental analysis, the direct and cot method are discussed and evaluated separately in the on-demand information extraction task. however, the potential synergistic effects of combining both methods have not been investigated. it could possibly yield insights into different dimensions of the task or further improve the model performance. 3. evaluation metrics: the evaluation metrics used in the current experiments are primarily detecting the overall similarity between the model outputs and the groundtruth. however, given the flexible nature of table structures, it is imperative to have evaluation metrics that can assess the accuracy and quality of table construction and information organization in a more fine-grained manner. developing more effective and precise evaluation metrics is necessary to robustly evaluate different aspects of our on-demand information extraction task. 4. contextual inference and complex instructions: as highlighted in the conclusion, the current model has room for improvement in contextually inferring table structures and processing complex instructions. this limitation can affect the utility and user experience, particularly for users who are not domain experts and may not know how to frame their queries optimally. enhancing the model's capabilities in these areas is essential for ensuring that the on-demand ie task is accessible and friendly for a wide range of users. in light of these limitations, future work should focus on exploring the impact of model size and data scale, investigating the combination of different data types, developing more nuanced evaluation metrics, and improving the model's ability to infer context and handle complex instructions. these efforts are crucial in advancing the on-demand information extraction systems and making them more accurate and widely applicable.