S ci C ap: Generating Captions for Scientific Figures (2021)

Abstract: Researchers use figures to communicate rich, complex information in scientific papers. The captions of these figures are critical to conveying effective messages. However, low-quality figure captions commonly occur in scientific articles and may decrease understanding. In this paper, we propose an end-to-end neural framework to automatically generate informative, high-quality captions for scientific figures. To this end, we introduce SCICAP, a large-scale figure-caption dataset based on computer science arXiv papers published between 2010 and 2020. After pre-processing – including figure-type classification, sub-figure identification, text normalization, and caption text selection – SCICAP contained more than two million figures extracted from over 290,000 papers. We then established baseline models that caption graph plots, the dominant (19.2%) figure type. The experimental results showed both opportunities and steep challenges of generating captions for scientific figures.

## introduction researchers use figures to explain complex concepts or show critical results. in scholarly articles, figure captions are critical to get the message across effectively. ones that are too generic (e.g., "results of experiment a.") or poorly written (e.g., "relations between x and y.") represent missed opportunities to explain scientific narratives to readers. unfortunately, such low-quality captions still occur in published scientific articles. this paper aims to develop automatic figure-captioning models that generate high-quality captions for figures and charts in scientific papers (figure 1 ). our motivation is two-fold. first, we aim to help researchers write better captions for the figures and charts in their papers. automatic caption models trained on informative, high-quality captions can suggest better captions. second, the proposed technology can make scientific charts and figures more accessible to blind or visually impaired readers. researchers have developed technologies to assist the blind to navigate graphical content, such as data visualization charts @xcite , printed physical maps @xcite , 3d chemical diagrams @xcite , and images on social media @xcite . however, only a few prior works focused on scientific figures. an image-captioning model specialized for scientific figures can improve the narration of scientific articles for the blind even when the original caption is unhelpful. to this end, we introduce scicap, a large-scale image-captioning dataset that contains real-world scientific figures and captions. scicap was constructed using computer science papers collected and released by arxiv. with pre-processing complete -including figure-type classification, subfigure identification, text normalization, and caption text selection -scicap contained more than two million figures extracted from over 290,000 papers. we then established baseline models that caption graph plots, the dominant (19.2%) figure type. the experimental results showed both exciting opportunities and steep challenges of generating captions for scientific figures. ## related work one of the few prior works attempting to caption scientific figures was by @xcite @xcite . they created figcap, a caption-figure pair corpus where the figures are synthesized, and used an lstm model with an attention mechanism to produce captions. figcap was built on research that aimed to analyze figure content automatically, including @xcite , fig-ureqa @xcite , and dvqa @xcite . dvqa and figureqa were both made using synthetic figures; figureseer contained over 60,000 figures across seven figure types extracted from research papers. meanwhile, @xcite proposed a set of "caption units" (such as title, label name, min/max, etc.) that are important to include in a caption of scientific figures; they created a model, figjam, to produce such units @xcite . also relevant is the "data-to-caption" work, which takes a chart's source data table and metadata as input to generate a caption @xcite . these models generate captions based on data tables, not the figures. ## constructing scicap dataset this section describes the process that massages real-world figure-caption data into an appropriate easy-to-use format for the nlp community. this data-processing procedure was developed iteratively and empirically. ## experimental results to examine the feasibility and challenges of creating an image-captioning model for scientific figures, we established several baselines and tested them using scicap. the caption quality was measured by bleu-4 @xcite , using the test set of the corresponding data collection as a reference. figure 2 shows some example outputs. baseline model. we used a classical imagecaptioning model, cnn+lstm architecture, as our baseline @xcite . the pre-trained resnet-101 @xcite was used as the image encoder to represent a figure as a 2048-dimension vector. this image vector was then fed into a dense layer to fit the dimension of the word-embedding and the lstm decoder where the word-embedding and lstm hidden layer size were all 512. a global attention mechanism was added to the lstm decoder to better model the context @xcite . the lstm decoder took the image vector as the initial state and generate captions. we designed three variations of the baseline models, vision-only, vision+text, and text-only. first sentence subfig filter norm. rule figsep b. a. #fig. vocab size bleu-4 416,804 30,776 .0259 352,719 24,355 .0236 12,666 .0224 133,543 11,946 .0219 single-sentence caption only subfig filter norm. rule figsep b. a. #fig. vocab size bleu-4 247,649 21,765 .0291 218,655 17,685 .0228 9,760 .0234 92,021 9,232 .0207 caption @xmath0. rule figsep b. a. #fig. vocab size bleu-4 395,024 37,885 .0231 341,350 30,316 .0098 15,642 .0173 132,120 14,974 .0172 the text information was the titles, legends, and x-y labels extracted from the figures (step 2 in section 3). another lstm was used as a text encoder to encode text information into a vector. for the vision+text variation, we concatenated the image vector and the text vector together and fed it into the lstm decoder for caption generation. the text-only variation only took the text vector as the feature for the lstm decoder. experimental setups. we trained the baseline models using an 80/10/10 train/val/test data split. the models were trained by minimizing a crossentropy loss with a doubly stochastic regularization @xcite using adam @xcite . the weights of the pretrained resnet-101 image encoder were partially frozen so that only convolutional blocks 2 through 4 were fine-tuned throughout the training process @xcite , 2020 ), (2) @xcite , and (3) @xcite .] data collection feature bleu-4 first sentence vision only .0219 vision+text .0205 text only .0213 single-sent caption vision only .0207 vision+text .0202 text only .0212 caption w/ <=100 words vision only .0172 vision+text .0168 text only .0165 results. we trained the models on each data collection with varying levels of data filtering and text normalization. effects of text normalization. our experiments did not show the clear benefits of normalizing text to the resulting bleu-4 scores. we will explore other methods to normalize text, for example, using advanced techniques to identify equations in text @xcite . ## conclusion and future work this paper introduces scicap, a large-scale imagecaptioning dataset that contains real-world scientific figures and captions. scicap was constructed using more than two million images from over 290,000 papers collected and released by arxiv. we also established several image-captioning baselines, showing the feasibility and challenges of generating captions for scientific figures. in the future, we will explore approaches to improve caption quality, such as taking advantage of large pre-trained language models @xcite , or using information in paper's full text to boost performance.