T ext2 DB : Integration-Aware Information Extraction with Large Language Model Agents (2024)

Abstract: The task of information extraction (IE) is to extract structured knowledge from text. However, it is often not straightforward to utilize IE output due to the mismatch between the IE ontology and the downstream application needs. We propose a new formulation of IE, Text2DB, that emphasizes the integration of IE output and the target database (or knowledge base). Given a user instruction, a document set, and a database, our task requires the model to update the database with values from the document set to satisfy the user instruction. This task requires understanding user instructions for what to extract and adapting to the given DB/KB schema for how to extract on the fly. To evaluate this new task, we introduce a new benchmark featuring common demands such as data infilling, row population, and column addition. In addition, we propose an LLM agent framework OPAL (Observe-Plan-Analyze LLM) which includes an Observer component that interacts with the database, the Planner component that generates a code-based plan with calls to IE models, and the Analyzer component that provides feedback regarding code quality before execution. Experiments show that OPAL can successfully adapt to diverse database schemas by generating different code plans and calling the required IE models. We also highlight difficult cases such as dealing with large databases with complex dependencies and extraction hallucination, which we believe deserve further investigation.

## the text2db task our task is defined over a set of (user instruction i, database b, documents d) instances. the goal is to automatically update the database (@xmath0. the user instruction i is a natural language sentence indicating the high-level scope and the type of operation. the database b contains multiple tables t that can be filled with pre-existing data entries. the database schema is available to the model, which outlines the data types, constraints, relationships, and integrity rules among different tables. unlike creating a database from scratch, our task focuses on enriching an existing database b with a collection of text documents d. this setting is more realistic but also more challenging since information to be extracted from d must be aligned with the schema and data in b. ## text2db benchmark our benchmark construction starts with selecting a set of databases b to work with, then finding relevant documents d and annotating instructions i and the updated databases b ′ . ## the opal framework we introduce the opal (observe-plan-analyze language model) framework, which starts with observing the target database, then generates the plan of action in code by referring to the database and ie tools and catches errors by both static and dynamic code analysis. the opal framework is an instance of an lm agent framework that interacts with a database environment, utilizes ie models as tools, executes actions as code, and improves its plan and tools using feedback from the database. ## related work llms for databases the majority of work involving the application of llms to databases focuses on the text-to-sql task @xcite @xcite which does not involve extraction. recently, llms with few-shot examples have shown to outperform fine-tuned smaller models @xcite @xcite . the structured view generation task in @xcite is closest to our text2db setting. however, their task only requires extraction from semi-structured documents, without the need for integration with existing tables. we note that when the target database is of the form of (entity name, entity attribute), our task resembles knowledge base population (kbp) @xcite . the key difference lies in the fact that our task requires dynamic adaptation to diverse databases. tool learning in llms tool learning, or function-calling, has emerged as a promising approach to extend the capability of large language models. in the tool learning paradigm, certain tools (such as internet search, a calculator, and image generation models) are provided to the llm. tool learning can be enabled by prompting with function definitions or specialized fine-tuning @xcite @xcite foot_0 . we refer the reader to @xcite for a comprehensive survey. in the opal framework, ie models act as external tools to the planner. unlike prior work that leaves all the work to the planner and assumes that tools as provided as-is, our observer provides essential insight into the database to support tool selection and selects demonstrations to assist the ie tools. ## conclusion and future work in this paper, we propose the new task text2db which updates a given database using values extracted from a document set following user instructions. text2db presents unique challenges in dynamic adaptation , extraction ambiguity and integrity requirements. we present a new benchmark and a new llm agent framework opal for this task. in particular, opal features 3 components: the planner, analyzer, and observer. we show that opal substantially improves update effectiveness over directly generating the plan, with significant gains coming from using demonstrations from the observer for ie models. at this point, we have only considered inserting values that do not previously exist into the database. however, conflict resolution has been a long-standing issue in the integration of databases, and we also foresee similar challenges in the integration of documents and databases. in this case, we need to consider the reliability of the document and the confidence of the extraction model.