Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLM s (2025)

Abstract: This paper defines and explores the design space for information extraction (IE) from layout-rich documents using large language models (LLMs). The three core challenges of layout-aware IE with LLMs are 1) data structuring, 2) model engagement, and 3) output refinement. Our study investigates the sub-problems and methods within these core challenges, such as input representation, chunking, prompting, selection of LLMs, and multimodal models. It examines the effect of different design choices through LayIE-LLM, a new, open-source, layout-aware IE test suite, benchmarking against traditional, fine-tuned IE models. The results on two IE datasets show that LLMs require adjustment of the IE pipeline to achieve competitive performance: the optimized configuration found with LayIE-LLM achieves 13.3–37.5 F1 points more than a general-practice baseline configuration using the same LLM. To find a well-working configuration, we develop a one-factor-at-a-time (OFAT) method that achieves near-optimal results. Our method is only 0.8–1.8 points lower than the best full factorial exploration with a fraction (~2.8%) of the required computation. Overall, we demonstrate that, if well-configured, general-purpose LLMs match the performance of specialized models, providing a cost-effective, finetuning-free alternative. Our test-suite is available at https://github.com/gayecolakoglu/LayIE-LLM

## introduction information extraction (ie) extracts structured data from unstructured documents, including layoutrich documents (lrds) as reports and presentations that mix visual and textual elements @xcite @xcite . lrds challenge traditional natural language processing (nlp) techniques, designed for plain texts @xcite . recent layout-aware models at the intersection of nlp and computer vision (cv) address this gap by including visual and structural features to improve ie from lrds (e.g., layoutlmv1-v3 @xcite , ernie-layout @xcite , graphdoc @xcite , docformer @xcite ). while early layout-aware models required substantial dataset-specific fine-tuning, recent generative document understanding models @xcite have demonstrated strong zero-and few-shot capabilities on unseen datasets and tasks. while llms exhibit significant potential, there exist open questions regarding their adoption for ie from lrds. in what form should the document content be provided to the llm? which methods are most effective for in-context learning (icl) and instruction tuning? do large multimodal llms as @xcite and qwen 2.5-vision @xcite offer advantages over traditional optical character recognition (ocr) and fine-tuned layout-aware models, and how does their performance vary across llms of different scales and sources? how coherent is the llm output, and what post-processing is needed? studies have individually addressed some of the aspects @xcite @xcite . however, there is a lack of a unified framework that comprehensively explores and compares these components. this paper fills this gap by systematically investigating the design space for llm-based ie from lrds. we focus on evaluating a range of preprocessing, chunking, prompting, and post-processing techniques, alongside various icl strategies, within a unified, reproducible framework. additionally, we conduct an extensive comparison of these components, examining multimodal llms, traditional models, open-and closed-source llms, and fine-tuned layout-aware models. to the best of our knowledge, no existing research has comprehensively compared the combined impact of these pipeline parameters across such diverse model categories. our results reveal multiple insights. first, current general llms can compete with traditional fine-tuned models such as layoutlmv3 and ernie-layout, without expensive labeling. second, rather than fine-tuning on data, llms require adjustment of the ie pipeline to achieve competitive performance-the performance gap between a general-practice baseline and the configuration tuned using our lightweight ofat method on two datasets is 13.3 f1 points for vrdu @xcite , and 37.5 for funsd @xcite , only 1.8 points and 0.8 points lower than the best possible brute-force configuration respectively. third, while purely text-based llms achieve competitive performance with our method, multimodal llms, those that directly integrate textual and visual features, still outperform them. however, this advantage comes with increased token usage, higher api costs, and diminished transparency and control over individual steps. in summary, our contributions are as follows. lrds using llms, addressing three core challenges: 1) data structuring, 2) model engagement, and 3) output refinement (sec. 2). ## layie-llm: test-suite for ie from lrds with llms we implement layie-llm , a comprehensive test suite to evaluate ie tasks from lrds. the pipeline, depicted in figure 2 , systematically transforms raw inputs into structured outputs through multiple stages, enabling a thorough assessment of the effectiveness of various design decisions. ## conclusion this paper introduces the design space for ie from lrds using llms, with challenges of data structuring, model engagement, and output refinement, considering the design choices within those challenges. we examine the effects of these choices on the overall performance through one-factor-ata-time (ofat) and full-factorial (brute-force) exploration of the parameters. we show that wellconfigured general-purpose llms provide an effective alternative to specialized models for ie from lrds. our cost-effective ofat method (2.8% of required computation) generalizes across two datasets (vrdu and funsd) with a performance that is just 0.8-1.8% lower than the best-possible configuration that can be found through a full-factorial exploration. this makes our results and method directly applicable to ie practitioners who need to configure their ie pipeline for their dataset. towards this goal, we have open-sourced layie-llm, our layout-aware ie test suite ( https: //github.com/gayecolakoglu/layie-llm ).