PDFT riage: Question Answering over Long, Structured Documents (2024)

Abstract: Large Language Models (LLMs) have issues with document question answering (QA) in situations where the document is unable to fit in the small context length of an LLM. To overcome this issue, most existing works focus on retrieving the relevant context from the document, representing them as plain text. However, documents such as PDFs, web pages, and presentations are naturally structured with different pages, tables, sections, and so on. Representing such structured documents as plain text is incongruous with the user’s mental model of these documents with rich structure. When a system has to query the document for context, this incongruity is brought to the fore, and seemingly trivial questions can trip up the QA system. To bridge this fundamental gap in handling structured documents, we propose an approach called PDFTriage that enables models to retrieve the context based on either structure or content. Our experiments demonstrate the effectiveness of the proposed PDFTriage-augmented models across several classes of questions where existing retrieval-augmented LLMs fail. To facilitate further research on this fundamental problem, we release our benchmark dataset consisting of 900+ human-generated questions over 80 structured documents from 10 different categories of question types for document QA. Our code and datasets will be released soon on Github.

## introduction when a document does not fit in the limited context window of an llm, different strategies can be deployed to fetch relevant context. current approaches often rely on a pre-retrieval step to fetch the relevant context from documents @xcite . these pre-retrieval steps tend to represent the document as plain text chunks, sharing some similarity with the user query and potentially containing the answer. however, many document types have rich structure, such as web pages, pdfs, presentations, and so on. for these structured documents, representing the document as plain text is often incongruous with the user's mental model of a structured document. this can lead to questions that, to users, may be trivially answerable, but fail with common/current approaches to document qa using llms. for instance, consider the following two questions: q1 "can you summarize the key takeaways from pages 5-7?" q2 "what year [in in the first question, document structure is explicitly referenced ("pages 5-7"). in the second question, document structure is implicitly referenced ("in we propose addressing this simplification of documents by allowing models to retrieve the context based on either structure or content. our approach, which we refer to as pdftriage, gives models access to metadata about the structure of the document. we leverage document structure by augmenting prompts with both document structure metadata and a set of model-callable retrieval functions over various types of structure. for example, we introduce the fetch_pages(pages: list[int]) function, which allows the model to fetch a list of pages. we show that by providing the structure and the ability to issue queries over that structure, pdftriage-augmented models can reliably answer several classes of questions that plain retrievalaugmented llms could not. in order to evaluate our approach, we construct a dataset of roughly 900 human-written questions over 90 documents, representing 10 different categories of questions that users might ask. those categories include "document structure questions", "table reasoning questions", and "trick questions", among several others. we will release the dataset of questions, documents, model answers, and annotator preferences. in addition, we release the code and prompts used. the key contributions of this paper are: the rest of the paper proceeds as follows: in section 2, we identify the related works to this one, and identify the distinguishing features of our work; in section 3 we outline the pdftriage approach, including the document representation, the new retrieval functions, and the prompting techniques; in section 4 we outline how we constructed the evaluation dataset of human-written questions; in section 5 we detail the experiments we run to support the above contributions; in section 6 we list the key takeaways of those experiments; and, lastly, in section 7 we describe the limitations of our current work and future directions. ## pdftriage: structured retrieval from document metadata the pdftriage approach consists of three steps to answer a user's question, shown in figure 1 : ## dataset construction to test the efficacy of pdftriage, we constructed a document-focused set of question-answering tasks. each task seeks to evaluate different aspects of document question-answering, analyzing reasoning across text, tables, and figures within a document. additionally, we wanted to create questions ranging from single-step answering on an individual document page to multi-step reasoning across the whole document. we collected questions using mechanical turk. 3 the goal of our question collection task was to collect real-world document-oriented questions for various professional settings. for our documents, we sampled 1000 documents from the common crawl to get visually-rich, professional documents from various domains, then subsampled 100 documents based on their reading level @xcite ). 4 by collecting a broad set of document-oriented questions, we built a robust set of tasks across industries for testing the pdftriage technique. in order to collect a diverse set of questions, we generated our taxonomy of question types and then proceeded to collect a stratified sample across the types in the taxonomy. each category highlights a different approach to document-oriented qa, covering multi-step reasoning that is not found in many other qa datasets. we asked annotators to read a document before writing a question. they were then tasked with writing a salient question in the specified category. for our taxonomy, we consider ten different categories along with their associated descriptions: table reasoning (7.4%): ask a question about a table in the document. 5. summarization (16.4%): ask for a summary of parts of the document or the full document. in total, our dataset consists of 908 questions across 82 documents. on average a document contains 4,257 tokens of text, connected to headers, subheaders, section paragraphs, captions, and more. in figure 2 , we present the document distribution by word count. we provide detailed descriptions and examples of each of the classes in the appendix. ## experiments we outline the models and strategies used in our approach along with our baselines for comparison. the code and datasets for reproducing our results will be released soon on github. ## results and analysis in in our annotation study, we asked the annotators to rank pdftriage compared to our two baselines, page retrieval and chunk retrieval (section 5). in figure 3 , we found that annotators favored the pdftriage answer over half of the time (50.7%) and favored the chunk retrieval approach over the page retrieval approach. when comparing different provided answers for the same question, pdftriage performs substantially better than current alternatives, ranking higher than the alternate approaches across all the question types. ## future work & conclusions in this work, we present pdftriage, a novel question-answering technique specialized for document-oriented tasks.