M at V i X : Multimodal Information Extraction from Visually Rich Articles (2025)

Abstract: Multimodal information extraction (MIE) is crucial for scientific literature, where valuable data is often spread across text, figures, and tables. In materials science, extracting structured information from research articles can accelerate the discovery of new materials. However, the multimodal nature and complex interconnections of scientific content present challenges for traditional text-based methods. We introduce MatViX, a benchmark consisting of 324 full-length research articles and 1,688 complex structured JSON files, carefully curated by domain experts in polymer nanocomposites and biodegradation. These JSON files are extracted from text, tables, and figures in full-length documents, providing a comprehensive challenge for MIE. We introduce a novel evaluation method to assess the accuracy of curve similarity and the alignment of hierarchical structures. Additionally, we benchmark vision-language models (VLMs) in a zero-shot manner, capable of processing long contexts and multimodal inputs. Our results demonstrate significant room for improvement in current models.

## introduction multimodal information extraction (mie) has become a key research focus, aiming to extract structured information from both text and visual content @xcite @xcite . this is particularly important in scientific literature, where valuable details are often spread across text, figures, and tables. the complex nature of scientific content, combined with the need to combine information from multiple sources, presents substantial challenges for traditional text-based extraction methods. in materials science, mie is crucial as research articles contain valuable data that can accelerate the discovery of new materials. tools like matrix component: dgeba epoxy resin filler name: batio3 filler name: 0.6 properties: [ { headers: frequency, loss tangent data: [1057.9, 27.2, …], … }, …] multimodal workflow structured extraction text: the epoxy resin (epr) as matrix was obtained… figure 1: example of a materials research article illustrating interconnected data between text and figures, with a json structure capturing sample properties and composition details. note that the matrix component is extracted from the text. gnome @xcite show how extracting structured databases from these publications can improve discovery efficiency. however, this process is complicated by the multimodal nature of scientific articles and the complex connections between data points. figures are particularly critical, as they often contain essential information about material properties not present in the text, making their accurate extraction vital for comprehensive information retrieval @xcite . recent methods like deplot @xcite tackle visual plot reasoning by converting plot images into linearized tables to enable pretrained llms to reason over visual data with minimal training. this approach allows llms to leverage few-shot reasoning for tasks like chart qa @xcite . however, deplot's focus on simple plot-to-table conversion makes it less suitable for the complex mie needed to handle interconnected data spanning long contexts, including text, tables, and multiple figures within full-length scientific documents. to address this gap, we introduce a novel benchmark, matvix, focused on obtaining structured information from materials science articles within the domains of polymer nanocomposites and polymer biodegradation. existing datasets such as funsd @xcite , cord @xcite , and kleister @xcite have made significant contributions to document analysis and information extraction, especially in handling complex layouts and long documents. however, they do not address the complexities of n -ary relation extraction, focusing instead on simpler tasks like named entity recognition (ner), which typically involves identifying predefined entities without capturing intricate relationships between them. furthermore, these datasets do not consider scientific documents, which often contain specialized language and complex figures. previous work @xcite in materials science has explored n -ary relation extraction, but primarily from text-only abstracts or short texts. pncextract @xcite represents a step forward by considering full-length articles, yet it remains limited to textbased content. in contrast, matvix considers all elements in long scientific documents, including text, tables, and figures, providing a comprehensive challenge for mie research (see previous work using pretrained models like layoutlm @xcite , layoutlmv2 @xcite , and domain-specific models such as matscibert @xcite has advanced the extraction of structured data from visually rich documents. however, these models are not suitable for our task, as extracting complex n -ary relations from long documents exceeds the token limit of 512 tokens in bert and layoutlm models. additionally, our task involves generating hierarchical json files, which is more complex than simple entity recognition or relation extraction and requires a sequence-to-sequence approach. splitting the documents to fit these models is not feasible, as it disrupts the extraction of interconnected relations across the entire document. matvix addresses the challenge of long document processing by utilizing vlms in a zeroshot manner. these models can process long input contexts (openai, 2023) without sacrificing accuracy and have demonstrated strong performance in tasks requiring both textual and visual reasoning. given their capabilities, one might wonder if we can simply input a scientific paper, complete with all its images, into a vlm to obtain structured data. in this paper, we benchmark these models and compare them against simpler baselines, showing that there is still substantial room for improvement. our task involves extracting complex hierarchical structures from scientific documents, where traditional evaluation metrics fall short. our method first aligns compositions, which serve as the identity of each sample, and then evaluates the properties represented by curves. to measure the closeness between predicted and ground truth curves, we use the fréchet distance, which captures how similar the overall trends are. our results show that while vlms show potential, significant improvements are needed. we also demonstrate that combining the best-performing vlms with the specialized deplot model enhances information extraction from figures. ## the matvix benchmark in this section, we first describe our dataset, including the problem definition and dataset preparation, and then explain our evaluation method for the task. matvix focuses on two critical domains: polymer nanocomposites (pnc) and polymer biodegradation (pbd). structured data in these fields is crucial for accelerating research and discovery, as it allows scientists to efficiently analyze relationships between material compositions and properties, which are often spread across text, figures, and tables within research articles. the emphasis on pnc and pbd reflects their significant representation within the field of materials science. the focus on pnc and pbd is justified by their significant presence in the field; a google scholar search yields approximately 95, 600 results for "materials science", 32, 100 for "polymer nanocomposites", and 17, 200 for "polymer biodegradation". each pnc and pbd sample in the dataset is represented as a structured json object that captures both the chemical composition and associated property data. a sample refers to a specific instance of a material with a defined chemical composition and measured properties. this structured format provides detailed information about the materials' compositions and includes the numerical property data needed for training machine learning models. these numerical data points are particularly important for developing models that can predict the relationship between material composition and performance. by leveraging these structured data representations, researchers can conduct large-scale analysis and modeling to advance material discovery and optimization @xcite . ## evaluation each paper contains a set of samples, and each sample is characterized by its composition and properties. the composition is represented as a set of strings, while the properties are captured as a list of curves. to evaluate the accuracy of the predicted samples against the ground truth, we follow a twostep process: first, we evaluate the alignment of the compositions within the samples, and then we assess the matching of the properties for the aligned samples. the reason for this approach is that the composition defines what the sample is, providing its identity, while the properties describe the characteristics of that specific sample. therefore, it is crucial to first match the compositions correctly, ensuring that we are comparing the same types of samples, before evaluating the properties within those matched samples. the evaluation employs the f1 score for composition alignment and two specialized metrics, the curve similarity score (css) and the curve alignment score (cas), for property evaluation. composition alignment to assess the alignment between predicted and ground truth compositions within each sample, we treat this as a maximum bipartite matching problem. each composition consists of a set of strings, and we aim to find the best correspondence between the predicted and ground truth compositions. we use the munkres algorithm (hungarian algorithm) @xcite to solve this bipartite matching problem, optimizing for the highest possible f1 score. if there are more ground truth samples for a paper than predicted samples, the unmatched ground truth samples are considered false negatives. conversely, if there are more predicted samples than ground truth samples, the unmatched predicted samples are considered false positives. curve similarity score (css) once compositions are aligned, we evaluate the properties within these matched samples. each property is represented by a curve, which is defined as a list of (x, y) points, and the css is introduced as a quantitative measure of similarity between the predicted and ground truth properties. these properties are typically plotted as the relationship between a variable and its corresponding response, such as how dielectric permittivity changes with frequency or temperature, which are commonly reported in research papers. the trends captured in these curves are crucial, as they convey important information about the material's behavior. therefore, accurately extracting and evaluating these curves is essential. to quantify the similarity, we use the levenshtein distance to compare the headers (x and yaxis labels) and the fréchet distance to compare the ground-truth data points of the curves. the fréchet distance measures the similarity between two curves by finding the smallest of the maximum pairwise distances. to compute this for polygonal curves, the discrete fréchet distance is used, as shown by @xcite , which involves determining the shortest path through a coupling sequence that matches points between the curves while maintaining their order. the css, given a predicted curve c p and a ground truth curve c t , is calculated as follows: this approach addresses several limitations inherent in the metric introduced in deplot, which misses critical information about the alignment of trends. by incorporating the fréchet distance, the css provides a more comprehensive evaluation, capturing both the trend similarities. ## benchmarking vlms the objective is to extract structured data from materials science documents. to achieve this, we first convert each pdf document into latex format using mathpix ( https://mathpix.com/ ). this approach generates a tex file that shows the structure of the paper, including sections, subsections, and all images. we then employ visual language models (vlms) in a zero-shot manner to extract structured json data. during our preliminary experiments, we observed that providing both the entire latex file and all associated images as input to the vlms leads to suboptimal results. additionally, since we must input images one at a time, it becomes costly to address this, we devised a multi-step pipeline (see appendix 5: formally, the steps can be described as follows: ŝimg,k i represents the expanded information obtained by the vlm for the kth image using ŝtext i as context. finally, ŝi merges the textual and image-based information to form a comprehensive structured dataset for document i. ## experiments in this section, we present the results of modeling with vlms on matvix. ## related work a plethora of prior works have devoted to unimodal information extraction (ie) with llms. see @xcite @xcite @xcite for an overview of their applications in scientific texts. in contrast, there lacks specialized ie systems that jointly operate on scientific documents that contain texts, tables, and images @xcite @xcite . general-purpose foundation models @xcite @xcite are appealing alternatives for such tasks; yet directly applying these models often yields subpar performance due to the complexity of document structures @xcite , their inability to reason over long contexts and/or multiple images @xcite , and performance differences across modalities @xcite . several recent works have endeavoured to adapt pre-trained foundation models for material science @xcite , but extending this fine-tuning approach to images has been challenging. this is attributed to subpar performance of open-weight vlms @xcite noted for their lack of faithfulness @xcite and compositionality @xcite compared to api-access models -as well as a lack of highquality multimodal datasets in the material science domain @xcite . matvix aims to bridge this gap by contributing an expertannotated, multimodal dataset over full-length scientific documents, and a workflow that achieves nontrivial performance according to a curated evaluation suite across different modalities. in this respect, matvix connects more broadly to a growing body of works that evaluate llms as agentic systems @xcite @xcite @xcite . compared to knowledge-intensive benchmarks, they arguably evaluate model capabilities more akin to daily workflows, and are robust against data contamination. matvix subsidizes this research with a dataset driven by scientific use cases, and offers a suite of partial evaluation metrics that enable users to identify areas of improvements compared to binary success metrics. ## conclusion and future work we introduce matvix, an expert-annotated, multimodal information extraction benchmark developed from scholarly articles. these articles receive a minimal amount of pre-processing; they are thus endowed with diverse textual, tabular, and visual structures, all of which contain information important for scientific applications. a general workflow is proposed, and is evaluated against a suite of automatic evaluation metrics that ensure the accuracies of extracted data across all modalities. results validate the performance of our workflow, and our automatic metrics agree with human evaluations. there are many avenues for future work. one such example is exploring an agentic framework where the model utilizes various smaller models or tools to assist with the extraction task. for instance, as shown in