Collage: Decomposable Rapid Prototyping for Co-Designed Information Extraction on Scientific PDF s (2025)

Abstract: Recent years in NLP have seen the continued development of domain-specific information extraction tools for scientific documents, alongside the release of increasingly multimodal pretrained language models. While applying and evaluating these new, general-purpose language model systems in specialized domains has never been easier, it remains difficult to compare them with models developed specifically for those domains, which tend to accept a narrower range of input formats, and are difficult to evaluate in the context of the original documents. Meanwhile, the general-purpose systems are often black-box and give little insight into preprocessing (like conversion to plain text or markdown) that can have significant downstream impact on their results.In this work, we present Collage, a tool intended to facilitate the co-design of information extraction systems on scientific PDFs between NLP developers and scientists by facilitating the rapid prototyping, visualization, and comparison of different information extraction models on scientific PDFs, regardless of their input modality. For scientists, Collage provides side-by-side visualization and comparison of multiple models of different input and output modalities in the context of the PDF content they are applied to; for developers, Collage allows the rapid deployment of new models by abstracting away PDF preprocessing and visualization into easily extensible software interfaces. Further, we enable both developers and scientists to inspect, debug, and better understand modeling pipelines by providing granular views of intermediate states of processing. We demonstrate our system in the context of information extraction to assist with literature review in materials science.

## introduction in recent years, systems based on large language models (llms) have broadened the public visibil-* equal contribution. figure 1 : collage allows users to inspect multiple models in different modalities by presenting a stage-by-stage, decomposed view of the pdf modeling pipeline. here, we see a pdf composed of text and tables, with entities from different models shown in red and yellow. the summary view shows extracted content, while annotations and inspection views allow the user to step back in the modeling pipeline ity of developments in nlp. with the advent of tools that have publicly accessible, user-friendly interfaces, experts in specialized domains outside nlp are empowered to use and evaluate these models inside their domains, for example to automatically mine insights from scientific literature. further, an increasing number of these tools are multimodal, handling not only text, but frequently images, or even pdfs directly. however, despite the accessibility of these tools, the processing pipelines they employ remain as end-to-end black boxes and provide little interpretability or debuggability in case of failure. further, these systems usually rely only on large, deployed models, potentially leaving other user priorities, such as interpretability, efficiency, or domain specialization, unaddressed. domain specific research in domains like clinical @xcite , legal @xcite , and scientific @xcite nlp have long histories. models in these areas remain less accessible; in order to run and evaluate these models on your own data, custom code is often needed. further, because many of these models are text-only, evaluating their results in the context of their eventual use -for example, directly on a pdf -poses a challenge. this paper presents collage, a tool that facilitates the rapid prototyping, visualization, and comparison, of multiple models across modalities on the contents of scientific pdf documents. collage was designed to address the interface between developers of nlp-based tools for scientific documents and the scientists who are the intended users of those tools. to address scientists' needs, we ground our design in a series of interviews with domain experts in multiple fields, with a particular focus on materials science. further, in cases where model results may not meet scientists' or developers' expectations, we visualize the intermediate representation at each step, giving the user a granular view of the modeling pipeline, allowing shared debugging processes between developers and users. collage is domain-agnostic, and can visualize any model that conforms to one of its three interfaces -for token classification models, text generation models, and image/text multimodal models. we provide implementations of these interfaces that allow the use of any huggingface token classifier, multiple llms, and several additional models without requiring users to write any code. all of the interfaces are easily implemented, and we provide instructions and reference implementations in our repository 1 . ## motivation collage is based on collected themes from interviews with 15 professionals across materials science, law, and policy, in which the authors ask about their practices for working with large collections of documents. for a reasonable scope, we focus on the 9 materials scientists in our sample, whose responses concern their process of literature review. we focus on three themes that emerged consistently from these interviews to inform our design of collage: varied focuses. one of the most prominent themes to emerge in our interviews is the variety of focuses that scientists, even in very closely related subfields, can have when reading a paper and information in tables. as pointed out above, many of our participants relied heavily on information provided in tables, rather than solely in the document text. as such, an important concern in the design of collage would be to allow multimodality in the models that it interfaces with and visualizes. older documents. our participants noted that they regularly work with documents across a wide time range. several participants noted that the work that they relied on most frequently were technical reports from the 1950s to the 1970s. these reports are now digitized, but are otherwise highly variable in their accessibility to modern processing tools: the ocr used when digitizing them can be inaccurate, they often contain noise in the scanned images, and layouts are less standardized. this can lead to confusion on whether issues with performance are the fault of models themselves, or preprocesing choices that cause that degraded performance. we therefore aim to provide an interface that allows users to inspect intermediate stages of processing, to better understand where a model may have failed, and what subsequent development should target next: whether better performing models, or better preprocessing. ## design and implementation we conceptualize our system in three parts: pdf representation, which parses and makes the content of pdfs easily available to downstream usage; modeling, i.e. applying multiple models to that pdf representation, backed by common software interfaces, which facilitate the rapid extension of the set of available models; and a frontend graphical interface that allows users to visualize and compare the results of those models on uploaded pdfs. we discuss the design choices and implementation details of each stage in the following subsections, and show an architectural overview in figure 2 . ## addressing needs from interviews our system is specifically designed to respond to the concerns raised in our interviews. first, to accommodate the varied processes of materials scientists, we design interfaces that allow for easy implementation of new models into our framework; our existing implementations of those interfaces also allow for the application of multiple llms and huggingface models directly in the context of the pdfs under review. this allows users to search for and evaluate models that suit their existing workflows. for tables, we both provide an interface and implementations that allow the comparison of proprietary and open-source table parsing systems. extending this work to new table models and evaluating them is simplified by our software and visualization interfaces. our inspection view is designed to address concerns about older pdfs: in being able to inspect the results of processing, users and engineers of this system can identify failtable transformer model with bounding boxes and parsed table (top), a huggingface transformer model with token-level tags (middle), and gpt-3.5 turbo, with json output parsed into a table (bottom). ## co-design with collage in collage, both options involve little more than implementing the llm call, so it's easy to do both and then compare. for the multimodal case, alice extends the image predictor interface, which allows her to receive as input the cropped image of any element on the page and pass that to an llm; for the text-only case, alice can easily access the underlying document representation use the already identified and parsed tables (which are in a dataframe-compatible format) and convert them into markdown for her linearization. she is able to return a dictionary in the same schema for both predictors, which will automatically be visualized in the frontend as a pandas dataframe. she commits her code, registers the predictors, and asks bob to take a look in the collage interface. in-context evaluation. bob then re-processes his paper through collage, making sure to check the boxes for alice's new table parsing predictors. in the summary view, he's able to compare the predicted, normalized tables to the original pdf, to verify that the models are performing the normalization correctly. he then picks the better performing model, and asks alice to create a pipeline that can process his entire dataset. alice is able to take the predictor, add it to the papermage recipe that underlies collage, and run it over bob's set of pdf documents, adding a step to export the parsed tables that bob saw in the collage interface. diagnosing errors. bob looks through the parsed tables from processing all of the pdfs, and notes that for the older pdfs, the parsed content doesn't look right. he'd like to diagnose the problem. because the processing that alice and bob run on these documents is the same as that underlying collage, the results can be visualized in the tool, even if they were not directly processed through it. bob loads the representation of the parsed older document, and is able to view the results from the model that didn't look right. while the bounding boxes for the table look correct in the annotations view, he's also able to see in the inspection view that the text detected within the table has not been correctly ocr'd. he can now contact alice to see if there's a fix for that problem, but in the meantime, he can examine the visualizations for his pdfs to understand how the publication year might affect whether the deployed suite of models can correctly extract and normalize information, and what the cutoff year might be for the results to be trustworthy. in this case, collage enables bob to self-serve cutting-edge nlp for his own use case, requiring that he involve alice only when collage's functionality needs extension. when that happens, bob and alice can both see results in the same interface, and can discuss errors and how to prioritize new work. when alice develops new predictors to address bob's needs, she is required to do no pdf processing or visualization, which are built into the tool, and bob can evaluate and compare the results of these new predictors in the same interface he's been using the whole time. for debugging, both bob and alice have access to the same representation and visualization as a shared source of truth, and collaborate to involve both nlp and subject matter expertise in how to fix the problem. collage can accelerate the process of collaboration between nlp developers and scientists, allowing for co-design and rapid prototyping with a shared representation. ## related work collage situates itself at the intersection of tools that offer reading assistance for scientific pdfs and tools that partially automate the process of literature review by means of information extraction. tools for scientific pdfs often focuses on interfaces that augment the existing pdf with new information, such as citation contexts @xcite , or highlights that aid skimming @xcite . however, most of these works are designed around and purpose-built for specific models. by contrast, collage draws from projects like papermage @xcite , by attempting to be model-agnostic, while at the same time providing a visual interface to prototype and evaluate those models. scientific information extraction and literature review automation also have long histories. collage's focus on materials science was driven by the field's existing investment into data-driven design @xcite , which focuses on using information extraction tools to build up knowledge graphs to inform future materials research. this adds to the existing body of work in chemical and material information extraction, including works like chemdataextractor @xcite and matscibert @xcite . works like @xcite showcase the growing interest in llm-based extraction; as llms increasingly become multimodal, this capability is likely to be used for tasks like scientific document understanding. while all of these tools are intended to be applied to documents from the materials science domain, they do not share an interface: most tools expect plain text, some, like chemdataextractor allow html and xml documents, and some work with images. collage aims to be a platform on which multiple competing approaches can be evaluated, regardless of the input and output formats they require. ## conclusion in this work, we present collage, a system designed to facilitate co-design and rapid prototyping of mixed modality information extraction on pdf content between scientists and nlp developers. we focus on a case study in the materials science domain, that allows materials scientists to evaluate models for their ability to assist in literature review. we intend for this work to be a platform on which to evaluate further modeling work in this area.