F ig E x: Aligned Extraction of Scientific Figures and Captions (2025)

Abstract: Automatic understanding of figures in scientific papers is challenging since they often contain subfigures and subcaptions in complex layouts. In this paper, we propose FigEx, a vision-language model to extract aligned pairs of subfigures and subcaptions from scientific papers. We also release BioSci-Fig, a curated dataset of 7,174 compound figures with annotated subfigure bounding boxes and aligned subcaptions. On BioSci-Fig, FigEx improves subfigure detection AP^b over Grounding DINO by 0.023 and boosts caption separation BLEU over Llama-2-13B by 0.465. The source code is available at: https://github.com/Huang-AI4Medicine-Lab/FigEx.

## introduction compound figure separation enables semantically consistent image-caption pairs for tasks like visual question answering and image captioning, yet vision-only pipelines still struggle with ambiguous panel borders and ignore caption text. each year, millions of scientific publications are published that include figures with captions that summarize protocols, quantitative results, and key findings. in the biomedical domain, over 60% of figures contain microscopy images, charts, heatmaps and annotations @xcite . consequently, treating a composite as a single image forces all subfigures to share one caption, violating the one-image-onecaption assumption and degrading downstream performance. captions for compound figures include both overall explanations and notes for each subfigure. to learn from these figures, we must separate each subfigure and align each panel with its corresponding caption segment. simple rules from the imageclef challenges used whitespace and grid layouts @xcite a h&e-stained… b analysis of… figure 1: figex separates a compound figure and caption into aligned subfigures and subcaptions. motivated by these insights, we introduce figex, a model guided by vision-language models (vlm) that separates compound figures and aligns each panel with its caption. this design first sends the image and full caption into the vlm to produce vlm-guided features and extract subcaptions. next, it feeds the image and those features into a deit @xcite to predict bounding boxes for each subfigure as in figure 1 . to connect these stages, figex injects a @xmath0. the hidden state of @xmath1, which outputs boxes. figex also uses an attention mechanism to highlight the correct caption snippet for each panel. existing benchmarks to evaluate these methods, including benchmarks such as imageclef biomedical @xcite , the medicat subfigure separation subset @xcite and recent yolo-ocr datasets @xcite , are restricted by either licenses or are too small to fully leverage large vlms. they are also limited to specific scientific fields such as medicine, where the majority of the subfigures could be fixed shapes with unique colored backgrounds (rectangle and black, for example, in x-rays, ct scans, microscopy images, etc.) to address these gaps, we created biosci-fig, a scientific figure separation dataset, containing 7,174 compound figures with 43,183 manually annotated subfigure bounding boxes. captions were split into fine-grained subcaptions using gpt-4.1 and then manually checked to ensure correct alignment, yielding an increase in publicly available compound figure separation data. our figex-7b model achieves strong subfigure detection performance compared to yolo11 @xcite , yolos @xcite and grounding dino @xcite baselines, and delivers superior caption separation quality relative to llama models @xcite . our contributions are summarized as follows: ## conclusion in this paper, we propose figex-7b, a compact vision-language model for aligned extraction of scientific figures and captions. the architecture combines a lightweight vit backbone with a visionlanguage model, showing that careful multimodal design can match or surpass much larger singlemodality systems for object detection and text separation tasks. to foster progress in compound figure analysis, we release biosci-fig, a curated dataset of 7,174 scientific figures with precise bounding boxes and aligned subcaptions, providing a new benchmark for fine-grained document understanding. our experiments reveal that vlm-guided spatial features and textual cues are complementary. in addition, moderate amounts of paired data enable multimodal fine-tuning to outperform larger models, and language guidance is especially valuable for rare or ambiguous panels.