Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization (2023)

Abstract: Good figure captions help paper readers understand complex scientific figures. Unfortunately, even published papers often have poorly written captions. Automatic caption generation could aid paper writers by providing good starting captions that can be refined for better quality. Prior work often treated figure caption generation as a vision-to-language task. In this paper, we show that it can be more effectively tackled as a text summarization task in scientific documents. We fine-tuned PEGASUS, a pre-trained abstractive summarization model, to specifically summarize figure-referencing paragraphs (e.g., “Figure 3 shows...”) into figure captions. Experiments on large-scale arXiv figures show that our method outperforms prior vision methods in both automatic and human evaluations. We further conducted an in-depth investigation focused on two key challenges: (i) the common presence of low-quality author-written captions and (ii) the lack of clear standards for good captions. Our code and data are available at: https://github.com/Crowd-AI-Lab/Generating-Figure-Captions-as-a-Text-Summarization-Task.

## introduction in scientific documents, effective figure captions help readers understand complex figures like bar charts, line charts, or pie charts. these captions describe the images and often include necessary context from the document's full text @xcite . unfortunately, even published papers often have poorly-written captions. as per our analysis (section 8.2), around 53.88% of line charts' captions in arxiv cs.cl papers are found to be unhelpful for nlp readers. automatic caption generation could aid paper writers by providing good starting captions that can be refined for better quality. previous research typically approached figure caption generation as a vision-to-language task, i.e., ## problem statement and terminology a document d contains n figures, f 1 to f n , where f i has a caption c i that was written by the document author. in document d, j sentences, m i,1 to m i,j , explicitly mention f i (e.g., "as shown in f i ..."). the objective of this work is to automatically generate a high-quality caption, c ′ i , for figure f i using only its mentions (m i,1 to m i,j ) and the surrounding text of the mentions in document d. in the rest of the paper, we use these terms: ## dataset before diving into our experiments and analyses, we first describe the dataset upon which our study is grounded. our results are based on a scientific figure caption dataset, scicap, and several preprocessing steps to fit it into our workflow. scicap is a dataset that contains over 416,000 line charts and captions extracted from more than 290,000 arxiv papers @xcite . it was one of the first large-scale figure-captioning datasets based on real-world scientific figures. however, it does not contain the paragraphs that mention the figure. to address this, we downloaded all the pdf files of the original arxiv papers used in sci-cap and extracted all the mentions and paragraphs as outlined in section 6.1. detailed information on preprocessing, including the dataset resplit and ocr extraction, are described in appendix b. ## motivating analysis to understand the correlation between mentions and captions, we performed a series of analyses using the data described in section 4. specifically, we investigated the extent to which the words in the figure captions are represented in the corresponding figure-mentioning paragraphs. we used awesome-align @xcite to obtain the alignment between the source texts (mentions, paragraphs, and ocrs) and captions. awesomealign compared the similarity of the words' contextual embeddings and assigned an alignment between words if the similarity passed a threshold. we used scibert @xcite to obtain contextual embeddings and softmax @xmath0.99 to reduce false alignments. after obtaining the alignments, we computed what percentage of information in the caption could be found in the source texts. the results shown in ## generating figure captions as a text summarization task figure 1 overviews the proposed pipeline. this section describes each step of the pipeline. ## experimental results a simple baseline: using extracted mentions as captions. motivated by our information overlap study (section 5), we created the reuse baselines. these baselines simply repurpose portions of the input text as the prediction. vision-to-language baselines. the vision-tolanguage generation treated this task as an imagecaptioning task that took the scientific figure image as input and generated a text to describe it. we compared two vision-to-language models as baselines. first, we built a sequence-to-sequence model by combining beit @xcite and gpt-2 @xcite . we also selected the trocr @xcite model, a transformer-based sequence-to-sequence model pre-trained for ocr tasks. compared to image encoders like vit @xcite and beit @xcite , which were trained on photos, ocr models trained on printed and handwritten documents align more closely with the scientific paper domain. all figures from scicap (106,391 training samples) were used for training since no mentions were required. ## in-depth analysis we conducted an in-depth investigation focused on two key challenges: (i) the common presence of low-quality author-written captions and (ii) the lack of clear standards for good captions. ## discussion is text really all you need? our results demonstrate that summarizing figure-mentioning paragraphs is sufficient to generate captions, as shown by the similar scores of pegasus p and pegasus p +o in what is the best length for captions? our research indicates that filtering shorter captions can facilitate the generation of more helpful captions. however, the resulting captions tend to be longer than usual, as shown in the pegasus p +o+b shift to the right in figure 4a . this raises a question: is it fair to compare short and long captions on usefulness, given that longer captions inherently contain more information? while our automatic evaluation addressed this by implementing length normalization, our human evaluations and quality annotations did not specifically instruct the annotators to consider caption lengths. nevertheless, we argue that even if we asked annotators to consider caption lengths while identifying helpful captions, the "ideal" caption length would differ among annotators due to multiple factors. for example, as shown in figure 4c , the length distributions of captions vary across domains. the low inter-agreement from our human evaluation (see section 7.2) also suggests that personal preferences could influence ideal caption length @xcite . moreover, the ideal length could also be dictated by the context: writers might favor shorter captions due to page constraints, while readers might prefer longer but informative ones @xcite . to tackle this issue, a potential future direction could be enabling models to generate captions of diverse lengths to suit different users and contexts. ## conclusion and future work this work presented a new perspective on automatic figure captioning, demonstrating that a language-based approach, i.e., summarizing figurereferring paragraphs, can outperform conventional vision-based methods. our analysis further showed many unhelpful captions in arxiv papers, highlighting data quality's impact on captioning performance. this work lays the groundwork for further research, including exploring new data selection, revision, and augmentation strategies to mitigate the effects of low-quality data, developing new evaluation methods, and creating more robust models that better handle noisy data. we also aim to expand the technology's scope to cover a wider variety of figures and article types.