Are Layout-Infused Language Models Robust to Layout Distribution Shifts? A Case Study with Scientific Documents (2023)

Abstract: Recent work has shown that infusing layout features into language models (LMs) improves processing of visually-rich documents such as scientific papers. Layout-infused LMs are often evaluated on documents with familiar layout features (e.g., papers from the same publisher), but in practice models encounter documents with unfamiliar distributions of layout features, such as new combinations of text sizes and styles, or new spatial configurations of textual elements. In this work we test whether layout-infused LMs are robust to layout distribution shifts. As a case study we use the task of scientific document structure recovery, segmenting a scientific paper into its structural categories (e.g., “title”, “caption”, “reference”). To emulate distribution shifts that occur in practice we re-partition the GROTOAP2 dataset. We find that under layout distribution shifts model performance degrades by up to 20 F1. Simple training strategies, such as increasing training diversity, can reduce this degradation by over 35% relative F1; however, models fail to reach in-distribution performance in any tested out-of-distribution conditions. This work highlights the need to consider layout distribution shifts during model evaluation, and presents a methodology for conducting such evaluations.

## introduction humans use layout to understand the organizational structure of visually-rich documents such as scientific papers, newspaper articles, and web pages. for instance, a reader might use fontsize and boldfacing to recognize a section title, while they might use spatial location to recognize a footnote. based on the intuition that layout aids in document understanding, recent work introduced layout-infused language models (lms). to improve document processing, these models incorporate layout features figure 1 : model performance on document structure recovery, comparing training and testing in-distribution vs out-of-distribution. error bars indicate standard deviation across runs. layout distribution shifts degrade model performance by up to 20 f1 (section 5.2). simple training strategies such as few-shot fine-tuning and increasing training diversity partially mitigate the drop shown here (section 5.3, section 5.4). with rising interest in processing visually-rich documents, some language models have been augmented with components specifically designed to process layout features (e.g., @xcite . layout-infused models accurately process documents with similar layouts to those seen during training @xcite , and can leverage visual information to better understand long-range dependencies @xcite . but in practice, models often encounter documents with different layouts -for instance, pages with a different number of columns, a different density of words on the page, and different locations of textual elements. in order to realistically evaluate model performance, we study model performance under layout distribution shifts. although robustness to text-distribution shifts has been relatively well-studied, layout distribution shifts pose unique challenges and may require unique solutions; therefore, we focus specifically on robustness to layout distribution shifts. here we present a case study to evaluate model robustness to layout distribution shifts. our case study focuses on the task of segmenting a scientific paper into its structural categories. for example, a scientific paper might be segmented into categories such as title, authors, caption, paragraph, and reference. we refer to this task as document structure recovery. 2 we chose to focus on this task because it serves as a testbed to determine whether layout-infusion remains beneficial under layout distribution shifts. document structure recovery requires grasping the organizational structure of a document, a key piece of information that layout conveys. moreover, layout-infused models have been shown to reach state-of-the-art performance on this task, in settings where the layout distribution is the same between training and testing @xcite . to test model robustness, evaluations must train and test on examples drawn from different layout distributions. however, existing datasets for document structure recovery use random train-test splits, and thus are ill-suited for evaluating model robustness. in this work we leverage publisher metadata to construct train-test splits that reflect layout distribution shifts. publisher metadata is a proxy for layout distribution shifts because publication venue is a key driver of layout differences -different publishers adhere to different style guides and templates (e.g., figure 2 ). moreover, layout differences across publishers reflect layout distribution shifts faced in practice as new publishers, templates, and style guides arise. we use publisher metadata to propose new train-test splits of an existing dataset (grotoap2, @xcite ) for scientific document structure recovery. these splits are designed to reflect layout distribution shifts; hence we refer to the splits as grotoap2-lds. using grotoap2-lds, we evaluate a set of layout-infused lms and find that model performance degrades by up to 20 f1 under layout distribution shifts (figure 1 ). we show that layoutinfused models can quickly adapt to new distributions, and that increasing training diversity can improve model robustness. however, even with diverse training sets and fewshot fine-tuning, performance on out-of-distribution layouts remains more than 2 f1 below in-distribution performance. although layout-infusion aids in processing documents with in-distribution layouts, layout-infused lms may overfit on features seen during training. we release our code and evaluation suite to enable future evaluations and to facilitate expansions of our evaluation suite. ## background and related work recent work established that models are often sensitive to distribution shifts. shifts in the distribution of text or image statistics have been shown to substantially degrade model performance (e.g., @xcite @xcite @xcite , even in cases when human performance is robust to these distribution shifts @xcite . for example, question-answering models struggle to generalize from text in wikipedia to text in newspaper articles @xcite , and image classification models struggle to generalize between images taken from different cameras @xcite . we extend this line of research to study robustness to shifts in the distribution of layout features. document structure recovery provides an opportune setting for evaluating robustness to layout distribution shifts. solving this task requires understanding how the text and visual layout of a page convey the organizational structure of the document, and layout-infused lms have been shown to reach near-human performance on this task @xcite . prior work has shown that models transfer poorly across different document types (e.g., from scientific papers to financial documents) @xcite . although different document types exhibit differences in layout, they also exhibit large differences in other features, such as the textual domain and the distribution of structural categories. it is therefore unclear whether poor transfer across document types is due to layout distribution shifts or other factors. in this work, we experiment using train-test splits exhibiting different layout distributions but with documents of the same type (i.e., scientific papers from biomedical journals). 3 existing evaluation datasets for document structure recovery include many document types, such as scientific papers @xcite @xcite , forms @xcite , receipts @xcite , and long-form business documents @xcite . we focus on scientific papers, where layout distribution shifts are prevalent (figure 2 ). although existing datasets for scientific document structure recovery contain documents with different layouts, existing train-test splits do not reflect layout distribution shifts. ## evaluation methodology to evaluate model robustness, we propose a set of new train-test splits of grotoap2. these splits reflect layout distribution shifts that occur in practice, and we refer to this set of splits as grotoap2-lds. in this section we formally define our task (section 3.1), describe our procedure for partitioning data into splits that emulate layout distribution shifts (section 3.2), and present a specific benchmark for evaluating robustness to layout distribution shifts (grotoap2-lds, section 3.3). ## results we use grotoap2-lds (section 3.3) to evaluate robustness to layout distribution shifts. for each experimental condition, models are evaluated on four test sets, each containing papers from a heldout layout distribution. unless indicated otherwise, model performance is reported as the average across these four test sets. ## conclusion this work studies whether layout-infused models are robust to layout distribution shift. we present a method for evaluating robustness to layout distribution shift, and construct grotoap2-lds, a new set of splits for the grotoap2 dataset that evaluate model robustness to layout distribution shifts. we use grotoap2-lds to evaluate a set of existing layout-infused models (layoutlm, lay-outlmv2, and layoutlmv3), and compare against two text-only models (bert, scibert). layout-infused models perform most accurately on documents with familiar layouts ( we hypothesize that layout-specific components overfit more because they receive less pretraining data compared to text-only components, or because they increase total model parameter count (e.g., layoutlm, layoutlmv2, and layoutlmv3 contain 20-45% more parameters than bert and scib-ert). 8 future work could test whether larger-scale pretraining improves robustness of layout-specific components. 8 we note that the discrepancy in generalization performance is not driven by the proportion of unk tokens. all tokenizers produced fewer than 0.3% unk tokens, and the tokenizer for scibert in fact had more unk tokens than the tokenizer for bert and the layoutlm models. we show that training strategies such as increasing training diversity or few-shot adaptation to the target layout distribution can mitigate the performance drop across layout distribution shifts. these results provide guidance for curating training data and highlight the importance during data collection of curating examples that reflect variation in document provenance. in situations with a known change in layout distribution (e.g., if a system trained on papers from one publisher is re-used to process papers from a new publisher), the cost of annotating a few examples from the target distribution may be highly effective, resulting in a large improvement in out-of-distribution model performance. this work highlights the importance of considering layout distribution shifts when evaluating models on tasks involving visually-rich documents such as scientific papers. we hope that our study and evaluation methodology facilitate the development of layout-infused models that can generalize across layout distribution shifts.