Schema-Driven Information Extraction from Heterogeneous Tables (2024)

Abstract: In this paper, we explore the question of whether large language models can support cost-efficient information extraction from tables. We introduce schema-driven information extraction, a new task that transforms tabular data into structured records following a human-authored schema. To assess various LLM’s capabilities on this task, we present a benchmark comprised of tables from four diverse domains: machine learning papers, chemistry literature, material science journals, and webpages. We use this collection of annotated tables to evaluate the ability of open-source and API-based language models to extract information from tables covering diverse domains and data formats. Our experiments demonstrate that surprisingly competitive performance can be achieved without requiring task-specific pipelines or labels, achieving F1 scores ranging from 74.2 to 96.1, while maintaining cost efficiency. Moreover, through detailed ablation studies and analyses, we investigate the factors contributing to model success and validate the practicality of distilling compact models to reduce API reliance.

## introduction vast quantities of experimental data are locked away in tables found in scientific literature. these tables are primarily designed for visual presentation, and the underlying data is typically not available in any structured format, such as a relational or graph database. some table collections have simple or uniform structures @xcite , making them easy to convert to relational data, for example, wikipedia tables @xcite , however a lot of information is stored in tables with complex and varied layouts, such as tables of results in papers found on arxiv.org. prior work on extracting data from tables has focused on developing custom pipelines for each 1 our code and data are available at https://github. com/bflashcp3f/schema-to-json . ## schema-driven information extraction we now describe schema-driven ie, a new task that extracts structured records from tables with minimal supervision. as shown in figure 1 , the task input contains two elements: 1) a table with numerous cells, optionally supplemented with contextual text, e.g., retrieved paragraphs from the same document; and 2) an extraction schema that outlines target attributes and their data types for various record types (implemented as json templates). given the input, the model generates a sequence of json objects, where each object corresponds to a cell in the table and contains key-value pairs for the pre-defined attributes of a specific record type. consider a table in an ml paper that displays various models' results. our proposed task enables the extraction of result records from each cell in the table. these records include relevant attributes such as the evaluation metric, task, etc, which are structured in corresponding json objects and could facilitate meta-analysis of experiments or support research on reproducibility. to demonstrate the feasibility of schema-driven ie on tables, we introduce instructe, a method to extract structured records from a broad range of semi-structured data, using only task-specific instructions. instructe uses a template-based approach to information extraction @xcite , where the extraction schema is represented as a series of json templates. the underlying llm is instructed to select the appropriate template and populate it with extracted values for each cell in an input table, following a specified cell traversal order. as illustrated in figure 2 (left), the prompt used by instructe consists of four key components: an input table (optionally) supplemented with contextual text, an extraction schema, task-specific instructions, and an initial record for starting the process. despite explicit instructions, we found that models often fail to generate json records for all the cells in a single inference pass. instead, models often deviate from the instructed cell traversal order, leading to partial extraction of the input table's cells. to mitigate this, we use an iterative error recovery strategy. as shown on the right side of figure 2 , we detect deviations from the instructed left-right, top-down order by comparing predicted cell values with those from a rule-based cell detector. then, we truncate the llm's output to the point of deviation, and re-prompt the model with the truncated sequence, adding the value of the next target cell. this process is repeated until all records are generated. using identified cells as a scaffold, this strategy helps the model adhere to prompt formulation in this work, we explore the task of intent classification … \begin{table}[!htbp] \begin{tabular}{@{}lllllll@{}} mode & \multicolumn{3}{l}{bus booking} \\ \midrule & 345m & 1.3b & 5b \\ zero shot & 0.755 & 0.762 & 0.787 \\ … \end{tabular} \caption{zero-shot and few shot (fs) performance… \end{table} input table (w/ supp. text) here are json templates for four types of numeric cells: "other", "result", "data stat.", and "hyper-param.": {"value": "xx", "type": "result", "task": "xx", … {"value": "xx", "type":"hyper-param.", "model": ... ... ## the schema-to-json benchmark we now present the details of our benchmark, schema-to-json, which is designed to assess the capabilities of llms to extract data from tables, adhering to a predefined schema. this benchmark contains tables from four domains: machine learning papers, chemistry literature, materials science journals, and webpages. each domain features a unique textual format, namely, l a t e x, xml, csv, and html, requiring models to generalize across domains and formats. for ml tables, we add relevant paragraphs from the same documents to provide additional context, testing the models' capacity to jointly understand tabular and textual data. we manually annotate datasets for the first two domains and adapt pre-existing datasets into our unified format for the latter two. statistics of the four datasets are summarized in pubmed chemistry tables we also annotate a new dataset of pubmed tables describing the physical properties of chemical compounds. the auto- table-f 1 calculated on double-annotated tables is used as the iaa score. a table-f 1 score of 91.0 underscores the reliability of the dataset. discomat @xcite we experiment with discomat, a dataset focusing on glass composition tables from elsevier material science journals. the task is to extract tuples comprising (material, constituent, percentage, unit) from given tables. we adapt discomat to fit our schema-driven ie framework by grounding the percentage element to numeric cells in the table and considering the other elements as attributes. the model is tasked to identify numeric cells representing constituent percentages and predict the associated three attributes. we refer readers to @xcite for more details of discomat. foot_3 swde @xcite finally, we add swde (structured web data extraction) as a fourth dataset, aimed at extracting pre-defined attributes from html webpages. this dataset com-prises roughly 124k pages gathered from eight distinct verticals, such as auto, book, and movie. each vertical includes ten unique websites and is associated with a set of 3 to 5 target attributes. for instance, the movie vertical seeks to extract attributes such as title, director, and genre. ## experiments we evaluate the capability of various llms to perform schema-driven ie, in addition to full finetuning using our benchmark. for ml and chemistry tables, we use a subset of 10 and 7 randomly sampled papers separately for model development, which facilitates the training of supervised baselines. for the two pre-existing datasets, we follow the data splits used in the original experiments. ## conclusion this paper explores the capabilities of llms for extracting structured data from heterogeneous tables. we introduce a new task, schema-driven information extraction, which converts tables into structured records guided by a human-authored data schema. to facilitate this task, we present a benchmark, comprised of tables from four diverse domains, and evaluate various llms through our proposed method instructe. the experiments reveal that while api-based models excel across domains and formats, open-source models display significant potential in specific areas. moreover, we conduct detailed ablation studies and analyses to investigate the factors for model success, and validate the feasibility of building compact models through distillation to reduce dependency on apis.