Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? (2024)

Abstract: Despite the remarkable capabilities of Large Language Models (LLMs) like GPT-4, producing complex, structured tabular data remains challenging. Our study assesses LLMs’ proficiency in structuring tables and introduces a novel fine-tuning method, cognizant of data structures, to bolster their performance. We unveil Struc-Bench, a comprehensive benchmark featuring prominent LLMs (GPT-NeoX-20B, GPT-3.5, GPT-4, and Vicuna), which spans text tables, HTML, and LaTeX formats. Our proposed FormatCoT aids in crafting format-specific instructions from the intended outputs to populate this benchmark. Addressing the gap in task-centered evaluation, we propose two innovative metrics, P-Score (Prompting Score) and H-Score (Heuristical Score), to more accurately gauge LLM performance. Our experiments show that applying our structure-aware fine-tuning to LLaMA-7B leads to substantial performance gains, outshining its LLM counterparts across most measures. In-depth error analysis and creating an ability map across six dimensions, coverage, formatting, reasoning, comprehension, pragmatics, and hallucination, highlight areas for future enhancements and suggest forthcoming research trajectories. Our code and models can be found at https://github.com/gersteinlab/Struc-Bench.

## introduction significant advancements have been made in various natural language processing tasks by large language models (llms) @xcite @xcite @xcite , especially in text generation tasks @xcite . the ability to output structured data, one of the key aspects of generative capability, has also attracted * contributed equally. despite their advanced capabilities, llms have problems with generating complex structured tables, an indispensable skill for practical applications like coding copilot and automated report generation. this proficiency entails the organization of information from multifarious sources into coherent structures. generating structured tables as outputs not only helps human understanding but also facilitates the automated data processing pipeline in autonomous language agents. furthermore, generating structured tables can also serve as a critical preprocessing procedure for downstream tasks such as decision-making and knowledge extraction. however, the current landscape of llm evaluation often neglects this aspect of table generation, which casts uncertainty on their full potential and utility in such scenarios. our research seeks to thoroughly investigate these gaps. first, there is a lack of systematic analysis and comprehensive benchmarks of the ability of llms to output complex structured tabular data. previous efforts on evaluating llms @xcite on structured data primarily centered around simple information extraction (ie) tasks: recognizing named entities, extracting relations, and detecting events. here the goal of ie tasks is to gather the extracted data in a highly structured form (zhong and chen, 2020). much earlier work was considerably more task-centric as opposed to llm-centric. the focus was predominantly on generating structured data from text (text-to-data) tasks with pre-trained models @xcite @xcite like bart @xcite and t5 @xcite . second, there is a lack of evaluation metrics of structured tabular data generation. existing benchmarks often rely on rudimentary objective metrics such as word overlap to measure the accuracy of dataset curation formatcot self-instruct with in-context examples train llama-7b guiding questions for prompting input: ###task: generate a latex table from given text ###text input: ###task: generate a latex table from given text and format description ###text ###format instruction ###data demo/examples:... ###describe the detailed format of a given latex table according to the commands and tags with more than 500 words whether there are table borderlines? how is text alignment? what are table attributes? whether to bold? whether to add \ref? whether there are horizontal and vertical lines bordering each row and column? say anything about special \" \ \" format token in latex. ## conclusion in summary, our study provides a thorough analysis of llms' challenges in structured table generation, introduces novel evaluation metrics, and assembles a specific benchmark covering a range of data types. we pinpoint key issues including content fidelity, format adherence, numerical reasoning, and management of extensive tables.