Document Structure in Long Document Transformers (2024)

Abstract: Long documents often exhibit structure with hierarchically organized elements of different functions, such as section headers and paragraphs. Despite the omnipresence of document structure, its role in natural language processing (NLP) remains opaque. Do long-document Transformer models acquire an internal representation of document structure during pre-training? How can structural information be communicated to a model after pre-training, and how does it influence downstream performance? To answer these questions, we develop a novel suite of probing tasks to assess structure-awareness of long-document Transformers, propose general-purpose structure infusion methods, and evaluate the effects of structure infusion on QASPER and Evidence Inference, two challenging long-document NLP tasks. Results on LED and LongT5 suggest that they acquire implicit understanding of document structure during pre-training, which can be further enhanced by structure infusion, leading to improved end-task performance. To foster research on the role of document structure in NLP modeling, we make our data and code publicly available.

## introduction long documents such as news articles, scientific papers, and clinical reports play a vital role in many human activities. these documents are usually organized into chapters, sections, subsections, and paragraphs, i.e. they are structured. this helps humans in navigating documents @xcite and building a mental model of the content @xcite . the example in fig. 1 shows how the hierarchy of sections and subsections helps when looking for the size of a dataset in an nlp ) -yet long texts exhibit structure, which helps in finding information @xcite . we investigate whether transformers learn representations of document structure during pre-training ( §4), whether structure-awareness can be enhanced by infusion after pre-training ( §5), and what effects infusion has on downstream task performance. source: qasper dataset, arxiv id 1909.00694 @xcite . although structure is omnipresent and useful to humans, existing long-document transformers (e.g. @xcite @xcite operate with linearized textual input: documents are converted to flat character strings, removing the distinction between different functional elements and their hierarchy (fig. 1 , top right). understanding the structural capabilities of longdocument transformers is important both theoretically and practically. from a theoretical standpoint, prior work in probing has demonstrated the ability of transformers to learn syntactic representations on the sentence level @xcite ) -yet little is known about the ability to induce higher-level discourse structures from linearized text. probing methodology and datasets for this investigation are missing. from a practical perspective, recent works demonstrate that structureaware modeling can improve downstream task performance @xcite @xcite ) -yet existing studies are limited to task-specific architectures and data formats, making it hard to generalize the findings to new tasks and document types. general-purpose methodology for communicating structural information to transformer models is yet to be established. our work aims to close this gap. instead of committing to a specific document format, we build the a task-and format-agnostic formalism of intertextual graphs @xcite to encode structure obtained from the original documents ( §3). building on this formalism, we investigate the role of document structure in long document transformers from two experimental angles: probing and downstream tasks. we introduce a novel suite of probing tasks in §4 to investigate structureawareness of pre-trained transformer models. probing experiments on two widely used long document transformer models -led @xcite and longt5 @xcite -suggest that transformers do acquire the ability to represent document structure during pre-training, but that there is room for improvement. consequently, in §5, we test the effect of adding structural information to the transformer input. we devise a generalpurpose structure infusion kit and employ it in experiments on our probing suite and two challenging long-document nlp datasets: qasper @xcite and evidence inference @xcite . the results suggest that structureawareness can be enhanced via infusion, leading to up to 6.8 f1 points increase on downstream tasks. our work lays the foundation for the systematic analysis of the role of document structure in long document modeling. long-document transformers. the memory and computational requirements of the standard transformer architecture @xcite scale quadratically with the input length, making it hard to process long documents under computational constraints. several innovations for increased efficiency have been proposed, surveyed by @xcite . a popular and well-performing approach is the combination of local attention with a varied distribution of global attention @xcite @xcite , used by the top 5 models in the scrolls benchmark for long-document processing @xcite . we experiment with two representatives for this approach: led @xcite , which is employed in many recent works on long documents (e.g. @xcite and longt5 @xcite , the best "base" model on the scrolls leaderboard at the time of writing foot_1 . ## infusing structure as exemplified in fig. 1 , abstract document structure can help humans in working with documents. while previous work shows that the addition of structural information can improve the downstream performance of transformer models @xcite @xcite , the use of task-specific architectures and document formats prevents comparison of structure infusion methods across the studies, and makes it challenging to relate performance to probing results. to remedy this, we introduce a task-and format-agnostic structure infusion kit, and demonstrate its wide applicability by studying the effects of structure infusion on led and longt5 and two challenging long-document tasks. ## conclusion in this work, we provided an in-depth analysis of the representation of abstract document structure in long-document transformers. experiments with our novel probing suite show that led and longt5 have learned to represent node function and hierarchical organization through pre-training without explicit supervision, with room for improvement. to investigate the effect of infusing the aspects of document structure that are missing in transformer inputs due to linearization, we developed a modular structure infusion framework. probing shows that structure infusion enhances the internal representation of document structure, and we see performance improvements from structure infusion on qasper and evidence inference, two downstream tasks where this has not been shown before. the significant correlation between several probing and downstream tasks suggests that it is indeed the improved representation of document structure that leads to downstream task performance gains. our probing, structure infusion and downstream task suite is easily extensible with new probing and downstream tasks and new types of infused information. while this work provides proof of the utility of our graph-based framework for documents from the scientific domain, the framework can be applied to other document types (e.g. web pages or conversation threads). given that the ad-dition of separator tokens between document elements can already increase performance, we deem applying our methods to documents with less welldefined structure promising. our probing methods are fully compatible with the current generation of transformer-based llms (bigscience @xcite , as long as the internal states of the model can be accessed. we hope that our contributions pave the path towards systematic study of the role of document structure in nlp.